CUDA error: an illegal memory access was encountered

[zhaojiancheng007]

Thanks for your great work; it's very interesting. When I was testing the code, I often encountered issues with illegal CUDA memory access during program execution. Have you encountered this problem? What could be causing this?" Thanks !

RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Xinjie-Q]

Could you provide the running command? When I run the commands from the script, I do not have such a problem.

[zhaojiancheng007]

Hi ! here is my running script

for num_points in 1000 1500 2000 do CUDA_VISIBLE_DEVICES=0 python train.py -d $data_path \ --data_name beauty --model_name GaussianImage_RS --num_points $num_points --iterations 30000 --save_imgs done

This error occored kind of random, but more frequent when I tested gaussians Image using relatively high resolution data, for example 1920 x 960, or 1280 x 960.

[ChaosAdmStudent]

This issue happens when you compile gsplat during first run (JIT). When I pip install gsplat by compiling CUDA libraries on installation, everything works for me. Maybe something to try out.

[shan1921]

Has this issue been resolved? If so, how was it resolved?