Open aiertamundarain opened 11 months ago
Hi @aiertamundarain, could you post the full error log? Thanks!
Hi @chenhsuanlin
This is the error log
Traceback (most recent call last):
File "train.py", line 104, in TORCH_USE_CUDA_DSA
to enable device-side assertions.
It doesn't look like the correct stack trace. Could you share the error log with CUDA_LAUNCH_BLOCKING=1
as suggested from the error message? (It would also be great if it could be formatted as code in the issue/comments!)
I have launch a train in neuralangelo docker ang raises the following error
RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with
TORCH_USE_CUDA_DSA
to enable device-side assertions.