RuntimeError: CUDA error: an illegal memory access was encountered

[GabrieleGalimberti-GaleSelector]

Hi, when I try demo.py in tools directory with a custom dataset I found this error after ~10/20 pointcloud. I'm not sure that is the size of pcd (*.npy) that is too big.

Command that I exec: python demo.py --cfg_file cfgs/kitti_models/PartA2.yaml --ckpt kitti_pretrained_models/PartA2_7940.pth --data_path media/dataset/

(25816, 4)
2021-04-18 16:54:53,469   INFO  Visualized sample index:    14
Traceback (most recent call last):
  File "demo.py", line 120, in <module>
    main()
  File "demo.py", line 98, in main
    pred_dicts, _ = model.forward(data_dict)
  File "/home/galimberti/projects/OpenPCDet/pcdet/models/detectors/PartA2_net.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/galimberti/projects/OpenPCDet/pcdet/models/roi_heads/partA2_head.py", line 200, in forward
    x_part = self.conv_part(part_features)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/spconv/modules.py", line 134, in forward
    input = module(input)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/spconv/modules.py", line 134, in forward
    input = module(input)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/spconv/conv.py", line 181, in forward
    use_hash=self.use_hash)
  File "/home/galimberti/.conda/envs/galimberti/lib/python3.6/site-packages/spconv/ops.py", line 95, in get_indice_pairs
    int(use_hash))
RuntimeError: CUDA error: an illegal memory access was encountered

Thanks for watching

[RolandoAvides]

Check if you can import spconv correctly.

[MartinHahner]

It's the same issue as here: https://github.com/open-mmlab/OpenPCDet/issues/338 There is no solution yet.

[GabrieleGalimberti-GaleSelector]

Thank you @MartinHahner

For other viewers: if you try to use it and you find this error at file k, you can check if the file is created in the output directory and pass beyond k input point clouds. Whit this method you restart input from k. I obtained every result whit this approach.

But I don't understand from which reference system are bounding box results. Can you help me?

[HuangVictorAuto]

Hey, Gabriele, I encountered the same problem when I try to test the pre-trained model on my own data. The demo works fine with the Kitti data, but fails with my own data. Have you solved the issue?

python demo.py --cfg_file cfgs/kitti_models/pv_rcnn.yaml --ckpt pv_rcnn_8369.pth --data_path test_00000.bin 2021-07-19 18:21:33,949 INFO -----------------Quick Demo of OpenPCDet------------------------- 2021-07-19 18:21:34,132 INFO Total number of samples: 1 2021-07-19 18:21:35,584 INFO ==> Loading parameters from checkpoint pv_rcnn_8369.pth to CPU 2021-07-19 18:21:36,150 INFO ==> Done (loaded 367/367) 2021-07-19 18:21:36,172 INFO Visualized sample index: 1 Traceback (most recent call last): File "demo.py", line 103, in main() File "demo.py", line 91, in main preddicts, = model.forward(data_dict) File "/home/victor/projects/pcdet/OpenPCDet/pcdet/models/detectors/pv_rcnn.py", line 11, in forward batch_dict = cur_module(batch_dict) File "/home/victor/anaconda3/envs/testpcd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, kwargs) File "/home/victor/projects/pcdet/OpenPCDet/pcdet/models/backbones_3d/pfe/voxel_set_abstraction.py", line 226, in forward features=batch_dict['multi_scale_3d_features'][src_name].features.contiguous(), File "/home/victor/anaconda3/envs/testpcd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, *kwargs) File "/home/victor/projects/pcdet/OpenPCDet/pcdet/ops/pointnet2/pointnet2_stack/pointnet2_modules.py", line 72, in forward xyz, xyz_batch_cnt, new_xyz, new_xyz_batch_cnt, features File "/home/victor/anaconda3/envs/testpcd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, kwargs) File "/home/victor/projects/pcdet/OpenPCDet/pcdet/ops/pointnet2/pointnet2_stack/pointnet2_utils.py", line 142, in forward grouped_xyz[empty_ball_mask] = 0 RuntimeError: CUDA error: an illegal memory access was encountered

[GabrieleGalimberti-GaleSelector]

I solve partially @HuangVictorAuto running multiple times the same program cheching if a pcd is processed. I skip pcd processed.

0 1 2 3 STOP

skip 0 skip 1 skip 2 skip 3 4 5 6 7 ...

I lost my configuration of packages: which config are you using for openpcdet? python, torch, ... please

[HuangVictorAuto]

my configuration are as followed: python 3.7,torch 1.7,cuda 11.0, cudnn 8.1, spconv 1.2.1,pcdet 0.3.0

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.