Inference on Custom dataset

[abhigoku10]

@tianweiy thanks for sharing the wonderful code base , i wanted to knw which file has to be executed to obtain inference on custom dataset , i have only the point clouds in bin file format which has x,y,z and intensity values. THnaks in advance !!

[abhigoku10]

@tianweiy can i use the default nusc config file for custom data or should i use "nusc_centerpoint_pp_02voxel_circle_nms_demo.py" which you have mentioned in single_inference.py

[tianweiy]

any configs should work

[abhigoku10]

Thanks !!

[tianweiy]

I will add a script next week. I think we want some script that basically turns a NumPy point cloud into a few numpy boxes. In this way, we can also do further optimization (fp16, tensorrt, bn/conv fusion, etc..).

[YoushaaMurhij]

@tianweiy, I tried model.half() as in simple_inference_waymo.py but i dot this error:

voxels = torch.tensor(voxels, dtype=torch.float32, device=self.device)
RuntimeError: CUDA error: device-side assert triggered

I think,It is related to the BatchNom .I moved the process to cpu to get this error:

 File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/centerpoint/CenterPoint/det3d/models/detectors/point_pillars.py", line 51, in forward
    x = self.extract_feat(data)
  File "/home/centerpoint/CenterPoint/det3d/models/detectors/point_pillars.py", line 25, in extract_feat
    data["features"], data["num_voxels"], data["coors"]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/centerpoint/CenterPoint/det3d/models/readers/pillar_encoder.py", line 151, in forward
    features = pfn(features)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/centerpoint/CenterPoint/det3d/models/readers/pillar_encoder.py", line 44, in forward
    x = self.norm(x.permute(0, 2, 1).contiguous()).permute(0, 2, 1).contiguous()
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/batchnorm.py", line 136, in forward
    self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 2058, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: "batch_norm" not implemented for 'Half'

Did you face such an issue?

[tianweiy]

Yeah there are some problems in the code for half. I will fix this today. It should not be moved to cpu

[tianweiy]

I think it should work now?

[YoushaaMurhij]

I think it should work now?

Thanks. Everything is running well!

[YoushaaMurhij]

I will add a script next week. I think we want some script that basically turns a NumPy point cloud into a few numpy boxes. In this way, we can also do further optimization (fp16, tensorrt, bn/conv fusion, etc..).

Yes the _voxelgenerater is the bottleneck for inference on light devices like (Jetson Xavier.. etc). I tried to reduce the input point cloud size. it does improve the inference speed but it is not always suitable. If this process is moved to GPU, there will be a noticeable improvement.

[tianweiy]

yes, the thing you need to do is just replace the current voxelization https://github.com/tianweiy/CenterPoint/blob/8ddae3082151aafdfaa0f5a7b852eb13c10c031b/tools/simple_inference_waymo.py#L58 with a faster one. In theory, it should only takes 2-4 ms for whatever large point clouds.

[YoushaaMurhij]

@tianweiy, when calling voxel_generator for the first time, it take very long time after that it comes back to normal. Can we get rid of this first iteration long delay?

[tianweiy]

I haven't met that but if it is really just the first iteration. You can try run this first iteration in the initialization ? And then it could work well?

[YoushaaMurhij]

I haven't met that but if it is really just the first iteration. You can try run this first iteration in the initialization ? And then it could work well?

I will try that. these are the times.

  network voxel_gen time cost: 0.7885196208953857
  network inference time cost: 0.03793001174926758
  total callback time:         0.8492271900177002
  network voxel_gen time cost: 0.001783609390258789
  network inference time cost: 0.025906085968017578
  total callback time:         0.03858590126037598
  network voxel_gen time cost: 0.0018002986907958984
  network inference time cost: 0.025873184204101562
  total callback time:         0.03880882263183594

[tianweiy]

which file are you using? The ros script? The ros script uses a python voxel generator which needs time to compile at the beginning to my knowledge. You can replace with the voxel generator in spconv like I did in simple_waymo_inference

[YoushaaMurhij]

Thanks for mentioning that! I got this Assertion Error when replacing the voxel generator:

  voxels, coords, num_points = self.voxel_generator.generate(self.points)
  File "/usr/local/lib/python3.8/dist-packages/spconv/utils/__init__.py", line 170, in generate
    res = points_to_voxel(points, self._voxel_size,
  File "/usr/local/lib/python3.8/dist-packages/spconv/utils/__init__.py", line 69, in points_to_voxel
    assert block_filtering is False
AssertionError

[tianweiy]

I saw this in the past. It is some version problem. It should be easily fixable. Please take a look at simple_waymo_inference and check spconv code at this line (the few arguments to the function). I can take a look tonight if you still have a problem with this.

[tianweiy]

@YoushaaMurhij Have you solved this? I think it is related to spconv version like there are two voxelizer in the spconv master branch, voxelizerv1, voxelizerv2, they have a little different APIs and then you may face this assert issue with one specific version if some arguments are not set correctly. You can also use the default voxelizer in this repo, but in my experiment, this cpu voxelizer in numpy is slower than the c++ one in spconv

[YoushaaMurhij]

No. I am facing a problem with the dependencies. This spconv commit seems only to run with cuda10. Have you tried the latest spconv commit?

[tianweiy]

yes, latest one should work (I tried this). Maybe use the voxelizerv2 ?

[tianweiy]

change this line https://github.com/tianweiy/CenterPoint/blob/9fdf572e854d20d58f4b0b8ef9f5785a5d39147a/tools/simple_inference_waymo.py#L58 to

        voxel_output = voxel_generator.generate(points)
        if isinstance(voxel_output, dict):
            voxels, coordinates, num_points = \
                voxel_output['voxels'], voxel_output['coordinates'], voxel_output['num_points_per_voxel']
        else:
            voxels, coordinates, num_points = voxel_output

[YoushaaMurhij]

Thanks. I tried it and it works! But it seems to work a little slower than the cpu voxelizer which is strange!

[YoushaaMurhij]

@tianweiy , It is not clear how the voxelizer chooses the points to include them in a certain voxel/pillar. Is it randomly adding the first max_points that are in this voxel/pillar? for pointpillars situation it will be a problem as the pillars height is 6 or 8 meters. for pedestrian detection the pillar will contains only some upper or lower points. Can you correct me please? Can you refer to the code that does this task? Thank you!

[tianweiy]

random, you can increase max_points

[tianweiy]

https://github.com/tianweiy/CenterPoint/blob/9fdf572e854d20d58f4b0b8ef9f5785a5d39147a/det3d/ops/point_cloud/point_cloud_ops.py#L112

[YoushaaMurhij]

In the first image the voxel takes the first n_max points and in the second one I shuffled the points. (The white points are the ones that are included in voxels while the colorful ones are the original pointcloud) when the pedestrian is very close to the lidar like in these images, the network is unable to detect it. Increasing the max_points in every voxel/pillar doesn't solve that.

[muzi2045]

maybe you need the dynamic voxel to process this problem.

[YoushaaMurhij]

maybe you need the dynamic voxel to process this problem.

@muzi2045 Is this "dynamic voxel" implemented somewhere here?

[muzi2045]

you can reference this : https://github.com/open-mmlab/mmdetection3d/tree/master/configs/dynamic_voxelization @YoushaaMurhij