The official PyTorch Implementation of RTM3D and KM3D for Monocular 3D Object Detection
MIT License
454
stars
85
forks
source link
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 32, 1]], which is output 0 of IndexPutBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). #52
Traceback (most recent call last):
File "./src/main.py", line 111, in
main(opt)
File "./src/main.py", line 73, in main
log_dicttrain, = trainer.train(epoch, train_loader)
File "/data/RTM3D/src/lib/trains/base_trainer.py", line 163, in train
return self.run_epoch('train', epoch, data_loader,unlabel_loader1,unlabel_loader2,unlabel_set,iter_num,uncert)
File "/data/RTM3D/src/lib/trains/base_trainer.py", line 116, in run_epoch
loss.backward()
File "/home/user/anaconda3/envs/rtm37/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/user/anaconda3/envs/rtm37/lib/python3.7/site-packages/torch/autograd/init.py", line 95, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 32, 1]], which is output 0 of IndexPutBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
how to solve it? i have no idea
Traceback (most recent call last): File "./src/main.py", line 111, in
main(opt)
File "./src/main.py", line 73, in main
log_dicttrain, = trainer.train(epoch, train_loader)
File "/data/RTM3D/src/lib/trains/base_trainer.py", line 163, in train
return self.run_epoch('train', epoch, data_loader,unlabel_loader1,unlabel_loader2,unlabel_set,iter_num,uncert)
File "/data/RTM3D/src/lib/trains/base_trainer.py", line 116, in run_epoch
loss.backward()
File "/home/user/anaconda3/envs/rtm37/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/user/anaconda3/envs/rtm37/lib/python3.7/site-packages/torch/autograd/init.py", line 95, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 32, 1]], which is output 0 of IndexPutBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).