search

ByChelsea / VAND-APRIL-GAN

[CVPR 2023 Workshop] VAND Challenge: 1st Place on Zero-shot AD and 4th Place on Few-shot AD
162 stars 20 forks source link

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: #9

Closed cclamd closed 10 months ago

cclamd commented 10 months ago

when run !python /content/VAND-APRIL-GAN/train.py --train_data_path "/content/VAND-APRIL-GAN/data" --config_path "/content/VAND-APRIL-GAN/open_clip/model_configs/ViT-B-16.json" it shows this error!

/usr/local/lib/python3.10/dist-packages/torch/autograd/init.py:200: UserWarning: Error detected in LinalgVectorNormBackward0. No forward pass information available. Enable detect anomaly during forward pass for more information. (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:92.) Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass Traceback (most recent call last): File "/content/VAND-APRIL-GAN/train.py", line 176, in train(args) File "/content/VAND-APRIL-GAN/train.py", line 140, in train loss.backward() File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/usr/local/lib/python3.10/dist-packages/torch/autograd/init.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1, 196, 512]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

nuclearliu commented 9 months ago

I have the same issue here

tangyz213 commented 4 months ago

Could you teach me how to resolve this problem？

oylz commented 2 months ago

fix, change this line from 

patch_tokens[layer] /= patch_tokens[layer].norm(dim=-1, keepdim=True)

to:

patch_tokens[layer] = patch_tokens[layer]/patch_tokens[layer].norm(dim=-1, keepdim=True)