search

skyhehe123 / SA-SSD

SA-SSD: Structure Aware Single-stage 3D Object Detection from Point Cloud (CVPR 2020)
492 stars 106 forks source link

RuntimeError: one of the variables needed for gradient computation has been modified #6

Open yangz0626 opened 4 years ago

yangz0626 commented 4 years ago

File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 19, in after_train_iter runner.outputs['loss'].backward() File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [11930]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I tried “inplace=False” but failed. How to solve it? Thanks!

AlphaPlusTT commented 4 years ago

i meet the same error when i use the pytorch1.0 and the error disappear when i use the pytorch1.1

AlphaPlusTT commented 4 years ago

@yangz0626 i found one inplace operation in ssd_rotate_head.py:327, maybe you can modify this line and have a try

dllucas commented 4 years ago

@AlphaPlusTT I try to use the pytorch 1.1, but there is something wrong with compiling spconv v1.1. However, I can not find spconv v1.0. Could you please help me?

zwqnju commented 4 years ago

@dllucas I follow the instruction in Det3D/INSTALLATION.md to install spconv-v1.0 ``

dllucas commented 4 years ago

@zwqnju Thank you~ I have also found that.

yangz0626 commented 4 years ago

@AlphaPlusTT thanks. It works when I change the pytorch version from 1.3 to 1.1