Closed hmm0852 closed 3 years ago
Hi @hmm0852 ,
You can comment out saved[-1][torch.isnan(saved[-1])] = 0
as it is mainly used for underflows and may throw an illegal memory access for some drivers.
Best, Alex
Will be closing this due to inactivity. Feel free to open a new issue in case of a different problem.
Best, Alex
Traceback (most recent call last): File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 251, in
main(args, logger)
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 171, in main
train(model, train_loader, optimizer, criterion, epoch, args, logger)
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/main.py", line 61, in train
loss.backward()
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
File "/home/mm/anaconda3/lib/python3.7/site-packages/torch/autograd/function.py", line 89, in apply
return self._forward_cls.backward(self, *args) # type: ignore
File "/media/mm/62f65e0e-b396-44ac-b1aa-b2fb260e70d1/mm/experiment/DE-Resnet18/softpool.py", line 45, in backward
saved[-1][torch.isnan(saved[-1])] = 0
My environment info is: GPU: Titan CUDA: 10.1.243 cuDNN: 7.6.5 PyTorch: 1.7.1