Closed kowo1001 closed 2 weeks ago
We have updated our code. Plz try it and let us know whether still exists some problems.
When I took the recently updated source code and ran it again, the following error occurred. I don't know why the error below occurs.
Any help is highly appreciated π
Thank U :D
I entered the command below.
(venv) jwjang@gpusystem:~/project/hsi_foundation/HyperSIGMA/ImageDenoising$ bash train_complex.sh 1e-4 hypersigma 2 l2 100
====================Command Result Start======================
mkdir: cannot create directory β./output/original_hypersigma_1e-4_spat-vit-base-ultra-checkpoint-1599_batch4_warmup_l2_epoch_100_complex_s3_8pointβ: File exists
Namespace(prefix='hypersigma_gaussian', arch='hypersigma', batchSize=4, lr=0.0001, wd=0, loss='l2', testdir=None, sigma=None, training_dataset_path='./dataset/WDC/training/wdc.db', pretrain='/mnt/code/users/yuchunmiao/hypersigma-master/pre_train/checkpoint-400.pth', init='kn', no_cuda=False, from_scratch=False, pretrain_path='./pre_train/spat-vit-base-ultra-checkpoint-1599.pth', no_log=False, threads=1, seed=2018, resume=False, no_ropt=False, chop=False, resumePath=None, dataroot='/data/HSI_Data/ICVL64_31.db', clip=1000000.0, gpu_ids=[0], basedir='./output/original_hypersigma_1e-4_spat-vit-base-ultra-checkpoint-1599_batch4_warmup_l2_epoch_100_complex_s3_8point', epoch=100, update_lr=5e-05, meta_lr=5e-05, n_way=1, k_spt=2, k_qry=5, task_num=16, update_step=5, update_step_test=10)
Cuda Acess: 1
=> creating model 'hypersigma'
load our vit fusion_new_v5 final models
Traceback (most recent call last):
File "/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/hsi_denoising_complex_wdc.py", line 23, in
It seems to be error about the loaded pretrain model. May I know ur env details, especially torch version? :D
It seems to be error about the loaded pretrain model. May I know ur env details, especially torch version? :D Thank U for ur reply :D
torch version: 2.3.1+cu121 python version: 3.10.12 and
Package Version
cafe 2.0.7 huggingface-hub 0.23.4 ipykernel 6.29.5 ipython 8.26.0 joblib 1.4.2 json5 0.9.25 jsonpatch 1.33 jsonpointer 3.0.0 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter_client 8.6.2 jupyter-console 6.6.3 jupyter_core 5.7.2 jupyter-events 0.10.0 jupyter-lsp 2.2.5 jupyter_server 2.14.1 jupyter_server_terminals 0.5.3 jupyterlab 4.2.3 jupyterlab_pygments 0.3.0 jupyterlab_server 2.27.2 jupyterlab_widgets 3.0.11 kiwisolver 1.4.5 lazy_loader 0.4 lmdb 1.5.1 matplotlib 3.9.0 matplotlib-inline 0.1.7 numpy 2.0.0 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.5.82 nvidia-nvtx-cu12 12.1.105 opencv-python 4.10.0.84 packaging 24.1 pandas 2.2.2 pillow 10.4.0 pip 24.1 protobuf 5.27.2 scikit-image 0.24.0 scikit-learn 1.5.1 scikit-plot 0.3.7 scipy 1.14.0 torch 2.3.1 torchlights 0.4.0 torchnet 0.0.4 torchstat 0.0.7 torchvision 0.18.1 urllib3 2.2.2 wheel 0.43.0
Do you need any other env details? I refer to the link below and it is said that there may be damage to the file itself. Is that correct?
https://wenku.csdn.net/answer/bd8127b290ba4812824b2d053cc5b58a
I have the same issue when loading the pre-trained models, @leonmakise.
@kowo1001 @aledamo Got it. Seem to be problems with the models we uploaded to huggingface. Let our team do a quick check. Some other models have similar error logs like this. Plz stay tune on it and give us some time. Thanks~
We found files are inconsistent with SHA256 check. A new model has been uploaded to huggingface. Please have a try and tell us if it is right now. https://huggingface.co/WHU-Sigma/HyperSIGMA/blob/main/spat-vit-base-ultra-checkpoint-1599.pth @kowo1001 @aledamo
We found files are inconsistent with SHA256 check. A new model has been uploaded to huggingface. Please have a try and tell us if it is right now. https://huggingface.co/WHU-Sigma/HyperSIGMA/blob/main/spat-vit-base-ultra-checkpoint-1599.pth @kowo1001 @aledamo
I tried again and the same error occurred. I think the same error occurs because of the problem below. Could you also change the spec-vit-base-ultra-checkpoint-1599.pth file and upload it back to huggingface? :D π
@kowo1001 Please check it now. https://huggingface.co/WHU-Sigma/HyperSIGMA/blob/main/spec-vit-base-ultra-checkpoint-1599.pth
I think the first question mentioned earlier will be resolved. Thank you. ππ However, there seems to be some issues regarding the second question.
Below is the content of the second question mentioned earlier. The input is a dc.tif file downloaded from the site below, and I created two mat files (train_0.mat, train_1.mat) using the create_WDC_dataset function of 'mat_data.py'. Then we tried to convert the two mat files (train_0.mat, train_1.mat) to the wdc.db file. I tried to create the data.mdb, lock.mdb, meta_info.txt files in the wdc.db folder through the createDCmall function of lmdb_data.py, but the meta_info.txt file was not created. Below is the related error.
https://engineering.purdue.edu/~biehl/MultiSpec/hyperspectral.html
(venv) jwjang@gpusystem:~/project/hsi_foundation/HyperSIGMA/ImageDenoising/utility$ python lmdb_data.py
====================Command Result Start======================
create wdc...
(926, 191, 64, 64)
map size (GB): 6.4770263671874995
/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/utility/WDC/wdc5.db
Traceback (most recent call last):
File "/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/utility/lmdb_data.py", line 95, in
@kowo1001 Please check it now. https://huggingface.co/WHU-Sigma/HyperSIGMA/blob/main/spec-vit-base-ultra-checkpoint-1599.pth
Thank U π
Hi @kowo1001 , we understand your concerns and have uploaded the training and testing data for you now. You can find them in https://huggingface.co/datasets/WHU-Sigma/HyperSIGMA_Datasets/tree/main/HyperSIGMA_denoising. Please have a try now.
Hi @kowo1001 , we understand your concerns and have uploaded the training and testing data for you now. You can find them in https://huggingface.co/datasets/WHU-Sigma/HyperSIGMA_Datasets/tree/main/HyperSIGMA_denoising. Please have a try now.
Thank you for your kindness. I wonder how you made the training dataset and the test dataset. Could you explain it in detail or give me a link to refer to? Thanks π
Hi @kowo1001, we are unsure about the second question mentioned earlier. You can refer to the code released in SST, which our data processing pipeline fully follows.
Hello, I have two questions.
(1) I entered the command below, but I don't know why the error below occurs. $ bash train_complex.sh 1e-4 hypersigma 2 l2 100
====================Command Result Start====================== Namespace(prefix='hypersigma_gaussian', arch='hypersigma', batchSize=4, lr=0.0001, wd=0, loss='l2', testdir=None, sigma=None, training_dataset_path='./dataset/WDC/training/wdc.db', pretrain='/mnt/code/users/yuchunmiao/hypersigma-master/pre_train/checkpoint-400.pth', init='kn', no_cuda=False, from_scratch=False, pretrain_path='./pre_train/spat-vit-base-ultra-checkpoint-1599.pth', no_log=False, threads=1, seed=2018, resume=False, no_ropt=False, chop=False, resumePath=None, dataroot='/data/HSI_Data/ICVL64_31.db', clip=1000000.0, gpu_ids=[0], basedir='./output/original_hypersigma_1e-4_spat-vit-base-ultra-checkpoint-1599_batch4_warmup_l2_epoch_100_gaussian_new_fusion', epoch=100, update_lr=5e-05, meta_lr=5e-05, n_way=1, k_spt=2, k_qry=5, task_num=16, update_step=5, update_step_test=10) Cuda Acess: 1 => creating model 'hypersigma' load our vit fusion_new_v5 final models Traceback (most recent call last): File "/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/hsi_denoising_gaussian_wdc.py", line 22, in
engine = Engine(opt)
File "/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/hsi_setup.py", line 146, in init
self.setup()
File "/data/jwjang/project/hsi_foundation/HyperSIGMA/ImageDenoising/hsi_setup.py", line 173, in setup
self.net = hypersigma(args=self.opt)
TypeError: spat_vit_b_rvsa() got an unexpected keyword argument 'args'
====================Command Result End======================
Any help is highly appreciated ;)
Thanks!
and, (2) I made the Washington dc mall.tif file( dc.tif ) into a train_0.mat, train_1.mat file through utility/mat_data.py in Image Denoising. Then I'm trying to make a train dataset. How can I make two mat files into a wdc.db file?
Or is there any other way to create a wbc.db file in training_dataset_path?
Please, suggest me how to do this.