xiuqhou / Salience-DETR

[CVPR 2024] Official implementation of the paper "Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement"
https://arxiv.org/abs/2403.16131
Apache License 2.0
140 stars 8 forks source link

训练的时候出现问题:RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation #30

Closed oneofforgettable closed 3 months ago

oneofforgettable commented 3 months ago

Question

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 8, 1092, 1092]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

补充信息

Traceback (most recent call last): File "main.py", line 222, in train() File "main.py", line 195, in train train_one_epoch_acc( File "/home/xx/DETR/Relation-DETR/util/engine.py", line 58, in train_one_epoch_acc accelerator.backward(losses) File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/accelerate/accelerator.py", line 2151, in backward loss.backward(**kwargs) File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass

xiuqhou commented 3 months ago

Hi,这个问题应该是Relation-DETR的bug,今天下午有其他研究者提出了相同的问题,链接在这里,我们下午修复了这个bug。您手头的代码可能不是最新版,请用最新版再试试看呢!

因为这不是本仓库Salience-DETR的问题,所以我先把issue关掉了哈,如果有问题可以在Relation-DETR的仓库开个issue~