search

pangsu0613 / CLOCs

CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
MIT License
348 stars 68 forks source link

Can CLOCs run on nuScenes dataset? #40

Closed JessieW0806 closed 3 years ago

JessieW0806 commented 3 years ago

1) Have you tested the performance of CLOCs on Nuscenes? Could you please share it with me？:) 2) For Nuscenes, there are more types of objects to be detected than KITTI. It can be thought that there will be uncountable 3D boxes and 2D boxes to calculate IOU (it is impossible to train a network for each object separately). In this case, how to transfer CLOCs to Nuscenes gracefully? 3) The reason for this problem is that I want to use millimeter-wave radar, which can add new modal information to CLOCS. Do you think this is feasible? Thanks a lot!!

pangsu0613 commented 3 years ago

Hello @JessieW0806.

  1. We have tested CLOCs on nuScenes, but we can not share it with you for now.
  2. nuScenes has 10 classes. CLOCs is a very small network (similar size to a detection head in other detectors), we simply use 10 separatee CLOCs networks for 10 different classes. If there are too many detection candidates for each class, you can do some simple filtering (thresholding) to reduce the amount, and based on our experience, this works well.
  3. I think it is feasible. Currently, CLOCs takes 3D bbox (from 3D detector, it could be LiDAR or other sensor based) and 2D bbox (from 2D detector, it could be camera or other sensor based). I guess millimeter-wave radar can only provide top-down view (BEV) measurements, such as object center points or BEV bounding boxes. There are some modifications you need to make to project these radar detections into the image and design a data association metric (for CLOCs, we use IoU) to associate these project radar detections and other 2D detections. I think this depends what detection format you have.