Open GXYM opened 4 years ago
The reason is that you run the code with GPU_ID != 0. If you use another GPU(1 or 2 or 3...), you should change GPU_ID in your project.
The reason is that you run the code with GPU_ID != 0. If you use another GPU(1 or 2 or 3...), you should change GPU_ID in your project.
The problem has solved.Although this problem is not caused by the GPU settings, I still want to thank you
THCudaCheck FAIL file=/build/python-pytorch/src/pytorch-1.1.0-opt-cuda/aten/src/THC/generic/THCTensorMath.cu line=16 error=700 : an illegal memory access was encountered Traceback (most recent call last): File "", line 89, in train loss.backward() File "/usr/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/usr/lib/python3.7/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 205, in
main()
File "", line 188, in main
train(model, train_loader, criterion, scheduler, optimizer, epoch, logger)
File "", line 93, in train
loss.backward()
File "/usr/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/lib/python3.7/site-packages/torch/autograd/init.py", line 87, in backward
grad_tensors = _make_grads(tensors, grad_tensors)
File "/usr/lib/python3.7/site-packages/torch/autograd/init.py", line 29, in _make_grads
new_grads.append(torch.ones_like(out))
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /build/python-pytorch/src/pytorch-1.1.0-opt-cuda/aten/src/THC/generic/THCTensorMath.cu:16