RuntimeError: CUDA error: an illegal memory access was encountered

[chetanmreddy]

Any idea why this might be happening?

`Starting initialization Downloading: "https://download.pytorch.org/models/resnet18-5c106cde.pth" to /root/.cache/torch/checkpoints/resnet18-5c106cde.pth Loading training dataset metadata:

• Can use 57874 images from the kitti (kitti_split) train set for depth training
• Can use 2975 images from the cityscapes train set for segmentation training Loading validation dataset metadata:
• Can use 1159 images from the kitti (kitti_split) validation set for depth validation
• Can use 500 images from the cityscapes validation set for segmentation validation Summary:
• Model name: sgdepth_chetan
• Logging directory: /cmudi001-sgd-1/SGDepth/Checkpoints/sgdepth_chetan_test/sgdepth_chetan
• Using device: cuda (GeForce GTX 1080 Ti) 100%|██████████| 46827520/46827520 [00:00<00:00, 57113495.00it/s] Epoch 0:
• kitti_kitti_train_depth losses at epoch 0 (batch 0):
  • avg 0.1202
• cityscapes_train_seg losses at epoch 0 (batch 0):
  • cross_entropy: 2.9558
• Breakdown of time spent this epoch:
  • unaccounted: 0.000
  • loading: 9.133
  • optimizer: 0.000
  • transfer: 0.075
  • forward: 3.802 THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered Traceback (most recent call last): File "train.py", line 372, in trainer.train() File "train.py", line 340, in train self._run_epoch() File "train.py", line 266, in _run_epoch loss.backward() File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: CUDA error: an illegal memory access was encountered`

[klingner]

Usually I do not get such errors, however cuda errors are somewhat cryptical in most cases. Did you change anything in the code with respect to the standard configuration?

Also: Did you already try to run the code on a CPU, just to verify that it is running? If you get an error on the CPU sometimes it is more readable than on the GPU. The code automatically detects if it can be run on GPU and if not available it runs on the CPU.

[chetanmreddy]

I did not change anything in the code except for correct paths to data.

I tried running on CPU and it trains without any errors. For now, I will keep this issue open and try to debug around. Thank You.

[chetanmreddy]

After following some solutions on PyTorch discussions forum, the error is changed to this:

Traceback (most recent call last): File "train.py", line 372, in trainer.train() File "train.py", line 340, in train self._run_epoch() File "train.py", line 249, in _run_epoch outputs = model(batch) File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, kwargs) File "/cmudi001-sgd-1/SGDepth/models/sgdepth.py", line 292, in forward x = self.seg(x) File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/cmudi001-sgd-1/SGDepth/models/sgdepth.py", line 79, in forward x = self.decoder(x) File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/cmudi001-sgd-1/SGDepth/models/networks/partialdecoder.py", line 134, in forward x = self.blocks[f'step{step}'](x) File "/opt/conda/envs/torch_110/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(input, kwargs) File "/cmudi001-sgd-1/SGDepth/models/networks/partial_decoder.py", line 69, in forward x_new = torch.cat((x_new, x_skp), 1) RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/THCCachingHostAllocator.cpp:278

UPDATE: It seems like the error is happening in the segmentation part. Can you please elaborate on how to prepare cityscapes dataset?