RuntimeError: CUDA error: an illegal memory access was encountered

[negiabhinit]

Checklist

• [X] I have searched for similar issues.
• [X] I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for main branch).

My Question

running the following script (o3dml) abhnegi@tghfg~/o3dml/Open3D-ML$ python scripts/run_pipeline.py torch -c ml3d/configs/pointpillars_kitti.yml --split test --dataset.dataset_path "/pfs/rdi/cea/rdicea_vru/01_Datasets/Kitti/" --pipeline ObjectDetection --dataset.use_cache True

gives me this error : Traceback (most recent call last): File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 261, in sys.exit(main()) File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 190, in main pipeline.run_test() File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 114, in run_test self.run_valid() File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 191, in run_valid results = model(data) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 132, in forward x = self.extract_feats(inputs) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 104, in extract_feats voxels, num_points, coors = self.voxelize(points) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 117, in voxelize res_voxels, res_coors, res_num_points = self.voxel_layer(res) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 361, in forward [torch.zeros_like(points_feats[0:1, :]), points_feats]) RuntimeError: CUDA error: an illegal memory access was encountered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.