RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 8, 1092, 1092]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
补充信息
Traceback (most recent call last):
File "main.py", line 222, in
train()
File "main.py", line 195, in train
train_one_epoch_acc(
File "/home/xx/DETR/Relation-DETR/util/engine.py", line 58, in train_one_epoch_acc
accelerator.backward(losses)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/accelerate/accelerator.py", line 2151, in backward
loss.backward(**kwargs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
Question
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 8, 1092, 1092]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
补充信息
Traceback (most recent call last): File "main.py", line 222, in
train()
File "main.py", line 195, in train
train_one_epoch_acc(
File "/home/xx/DETR/Relation-DETR/util/engine.py", line 58, in train_one_epoch_acc
accelerator.backward(losses)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/accelerate/accelerator.py", line 2151, in backward
loss.backward(**kwargs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass