Autograd error: In-place variable operation

[sbsky]

I'm attempting to train the network, but unfortunately it's raising an in-place operation on a variable required for gradient computation error:

  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/train.py", line 176, in <module>
    main()
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/train.py", line 114, in main
    losses, coarse_result, inpainted_result = trainer(x, mask, ground_truth)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/scripts/trainer.py", line 57, in forward
    refine_real, refine_fake = self.dis_forward(self.globalD, ground_truth, x2_inpaint)
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/scripts/trainer.py", line 69, in dis_forward
    batch_output = netD(batch_data)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/model/network.py", line 226, in forward
    x = self.linear(x)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py", line 1610, in linear
    ret = torch.addmm(bias, input, weight.t())
 (print_stack at ../torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
2020-07-07 08:23:49,820 ERROR one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16384, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/train.py", line 176, in <module>
    main()
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/train.py", line 172, in main
    raise e
  File "/Users/myuser/Documents/colab/Global-and-Local-Attention-Based-Free-Form-Image-Inpainting/train.py", line 130, in main
    losses['g'].backward()
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/Users/myuser/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16384, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Any ideas?

Edit: I'm using torch 1.5

[SayedNadim]

Hi, Can you try with PyTorch 1.4? I think this is because of version mismatch. You can refer to this issue. Cheers!

[sbsky]

Thanks, that seems to work. Any idea what could be the problem with torch 1.5?

Edit: sorry, getting my GitHub accounts mixed up

[SayedNadim]

Hi, I have not investigated the issue with version 1.5 yet. My first guess would be leaky relu inplace operation. I will update after experiments. Cheers!