CUDA error: an illegal memory access was encountered

[XuGW-Kevin]

While I have successfully run ns-train garfield --data /your/data/here and got the GARField's affinity field, the experiments fail when I try to run ns-train garfield-gauss --data /your/data/here --pipeline.garfield-ckpt outputs/your/data/garfield/.../config.yml. Namely, I can use the Global clustering function, but when I click on an instance and use Crop to click, the following error occurs:

Exception in thread Thread-3: Traceback (most recent call last): File "/home/%/miniconda3/envs/garfield2/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/home/%/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 218, in run outputs = self._render_img(action.camera_state) File "/home/%/nerfstudio/nerfstudio/viewer/render_state_machine.py", line 164, in _render_img outputs = self.viewer.get_model().get_outputs_for_camera(camera, obb_box=obb) File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/%/nerfstudio/nerfstudio/models/splatfacto.py", line 866, in get_outputs_for_camera outs = self.get_outputs(camera.to(self.device)) File "/home/%/nerfstudio/nerfstudio/models/splatfacto.py", line 766, in get_outputs depth_im = rasterize_gaussians( # type: ignore File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/rasterize.py", line 67, in rasterize_gaussians return _RasterizeGaussians.apply( File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(args, **kwargs) # type: ignore[misc] File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/rasterize.py", line 108, in forward num_intersects, cum_tiles_hit = compute_cumulative_intersects(num_tiles_hit) File "/home/%/miniconda3/envs/garfield2/lib/python3.8/site-packages/gsplat/utils.py", line 117, in compute_cumulative_intersects num_intersects = cum_tiles_hit[-1].item() RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Every time, the same error occurs. I encountered this while using CUDA 11.7 on an NVIDIA A40 GPU.

[kerrj]

What version of gsplat are you using? if you pip install --upgrade gsplat this issue might be fixed in a newer version.

[XuGW-Kevin]

This issue is solved after I upgraded CUDA to 12.2. Thanks!