search

loeweX / Greedy_InfoMax

Code for the paper: Putting An End to End-to-End: Gradient-Isolated Learning of Representations
https://arxiv.org/abs/1905.11786
MIT License
284 stars 36 forks source link

Failure to compute gradient #10

Closed rschwarz15 closed 4 years ago

rschwarz15 commented 4 years ago

Hi,

I have found your paper and code extremely interesting!

I am trying to run the vision training, but am coming across an error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 256, 1, 1]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

When I do torch.autograd.set_detect_anomaly(True) the issue is traced to line 57 in InfoNCE_Loss:

https://github.com/loeweX/Greedy_InfoMax/blob/8f91dc27fcc6edf1f5b9f005a9f5566bb796dce2/GreedyInfoMax/vision/models/InfoNCE_Loss.py#L55-L57

Any idea why this is hapenning?

loeweX commented 4 years ago

Hi,

Thank you for your interest in our work!

I haven't encountered that error before, so I cannot give a definite answer. My guess would be that it could have something to do with the pytorch version that you are using. Did you make sure to use version 1.0.0?

Alternatively, have you tried following the advice in this article?

rschwarz15 commented 4 years ago

Right, I'd had a conflict error in the environment set up and had changed it to PyTorch 1.5