Closed zyongbei closed 3 years ago
Would you mind providing:
So we can have a look at the problem.
Thank you.
Would you mind providing:
- Which dataset you were training on and the configuration file you were using?
- The version of Pytorch and Higher you were using
So we can have a look at the problem.
Thank you.
Hello, Configuration file: ./config/CIFAR10_LT/balms_imba200.yaml The version of Pytorch is 1.7.0 and Higher is 0.2.1 Thank you ~
Please downgrade your Pytorch to 1.4 for now, which is the version we used for testing. We will follow up on this issue.
Setting num_workers=0
in the config will solve the issue for Pytorch>=1.5.
The sample rates used in the backward function are a few iterations earlier than the current learner paramters since the data loader buffers some batches in sub-processes to accelerate training. The mismatch will be raised as a runtime error in Pytorch >= 1.5. Moving data loading to the main process solves the issue, meanwhile slows down training. Since the mismatch will not make a visible difference to the training outcome, using Pytorch<1.5 should be a more ideal option.
I will close the issue for now, feel free to reopen it if you have further questions.
https://github.com/jiawei-ren/BalancedMetaSoftmax-Classification/blob/c41238d1fa5fd5a27cf2b49d5398d8906bed7813/run_networks.py#L284
Configuration file: ./config/CIFAR10_LT/balms_imba200.yaml The version of Pytorch is 1.7.0 and Higher is 0.2.1
Hello, this error occurred when the meta_forward() function was run for the second time. I carefully review the code, but I don't know where the problem is.
Thankyou~
Traceback (most recent call last): File "main.py", line 161, in
training_model.train()
File "/data/byz/jupyter/BalancedMetaSoftmax-Classification/run_networks.py", line 360, in train
self.meta_forward(inputs, labels, verbose=step % self.training_opt['display_step'] == 0)
File "/data/byz/jupyter/BalancedMetaSoftmax-Classification/run_networks.py", line 288, in meta_forward
val_loss.backward()
File "/home/wangshuogroup/.conda/envs/open-mmlab/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/wangshuogroup/.conda/envs/open-mmlab/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [10, 1]], which is output 0 of TBackward, is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!