Hello I have one weird issue I cannot solve. I have my dataloader and dataset. When I train for lets say 2 images everything is fine, but when I train for >5 images, I always get this error:
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
If I write in terminal export CUDA_LAUNCH_BLOCKING=1 then I get error:
RuntimeError: CUDA error: an illegal memory access was encountered
and it reports there is some error in radii>1 in renderer file.
If I remove this radii>1 and just put some nonsense value like 1 then it reports same error on some other place. That's why I cannot figure it out what the error is.
Is it possible that reason is some cuda synchronization or out of memory (which would be weird if it is)?
robofar
commented
2 months ago
Hello I have one weird issue I cannot solve. I have my dataloader and dataset. When I train for lets say 2 images everything is fine, but when I train for >5 images, I always get this error:
RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
If I write in terminal
export CUDA_LAUNCH_BLOCKING=1then I get error: RuntimeError: CUDA error: an illegal memory access was encountered and it reports there is some error inradii>1in renderer file. If I remove thisradii>1and just put some nonsense value like1then it reports same error on some other place. That's why I cannot figure it out what the error is.Is it possible that reason is some cuda synchronization or out of memory (which would be weird if it is)?
Thanks