Testing Lidar failure (Inference without Lidar data)

[BeMuCa]

Hey Guys, I am trying to test the robustness of BEVFusion in case of complete Lidar Failure. For that I tried different approaches: First I tried working with different Lidar files, since the loading function tests for the existance of the pcd.bin files (with exactly the nuscenes names)

• I created files where every point is at 0 0 0 0 -> gave me (0.008 mAP).
• However, thats different than failure, then I tried empty files (np.array([])) ->

[                                                  ] 0/6019, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/test.py", line 232, in <module>
    main()
  File "tools/test.py", line 205, in main
    outputs = multi_gpu_test(model, data_loader, args.tmpdir, args.gpu_collect)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/mmdet/apis/test.py", line 98, in multi_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
    output = old_func(*new_args, **new_kwargs)
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/models/fusion_models/bevfusion.py", line 253, in forward
    outputs = self.forward_single(
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
    output = old_func(*new_args, **new_kwargs)
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/models/fusion_models/bevfusion.py", line 319, in forward_single
    feature = self.extract_features(points, sensor)
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/models/fusion_models/bevfusion.py", line 152, in extract_features
    feats, coords, sizes = self.voxelize(x, sensor)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 214, in new_func
    output = old_func(*new_args, **new_kwargs)
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/models/fusion_models/bevfusion.py", line 174, in voxelize
    ret = self.encoders[sensor]["voxelize"](res)
  File "/home/berk/.conda/envs/BEVFusion/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/ops/voxel/voxelize.py", line 131, in forward
    return voxelization(
  File "/home/berk/PycharmProjects/math-879-23-berk-hybrid-fusion/DeepFusion/BEVFusion/mmdet3d/ops/voxel/voxelize.py", line 55, in forward
    voxel_num = hard_voxelize(
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

With this error I realized I had to probably change the config file to just evade these errors:

• Tried changing configs/nuscenes/default.yaml:

  
  point_cloud_range: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  
  input_modality:
  use_lidar: false

Now I am kinda stuck trying to find a way to achieve this. 

I would really appreciate it if anyone has an idea on how to do this.

Thanks in advance!

[EpicGilgamesh]

Hey! what you did to fix it?

[BeMuCa]

I played around in the config files resulting in 0.52 mAP for no cam and ,as far as I remember, 0.006 mAP for no LiDAR