search

ruotianluo / self-critical.pytorch

Unofficial pytorch implementation for Self-critical Sequence Training for Image Captioning. and others.
MIT License
991 stars 278 forks source link

RuntimeError: Function MmBackward returned an invalid gradient #256

Closed TungWg closed 3 years ago

TungWg commented 3 years ago

Hi ruotianluo! I got an error when I use the command: $ python tools/train.py --id fc --caption_model newfc --input_json data/cocotalk.json --input_fc_dir data/cocotalk_fc --input_att_dir data/cocotalk_att --input_label_h5 data/cocotalk_label.h5 --batch_size 10 --learning_rate 5e-4 --learning_rate_decay_start 0 --scheduled_sampling_start 0 --checkpoint_path log_fc --save_checkpoint_every 6000 --val_images_use 5000 --max_epochs 30

error I got: Traceback (most recent call last): File "/home/wyd/ImgCaption/ImageCaptioning.pytorch-master/tools/train.py", line 194, in train loss.backward() File "/home/wyd/.conda/envs/pytorch/lib/python3.9/site-packages/torch/_tensor.py", line 255, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/wyd/.conda/envs/pytorch/lib/python3.9/site-packages/torch/autograd/init.py", line 147, in backward Variable._execution_engine.run_backward( RuntimeError: Function MmBackward returned an invalid gradient at index 0 - got [1, 512] but expected shape compatible with [1, 2560]

Sorry to bother you, but how can I sovle the problem? thanks!

ruotianluo commented 3 years ago

I clone the master branch and can properly run with this command. Did you change anything in the code?

TungWg commented 3 years ago

Thanks for your reply! I solved the problem. This seems to be some library version mismatch, I used pytorch1.9, then created a new environment with pytorch1.4, and it worked!

ruotianluo commented 3 years ago

Interesting... I was using pth1.9 too..... Wierd.