search

WeijingShi / Point-GNN

Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud, CVPR 2020.
MIT License
552 stars 114 forks source link

Queries-Training,Testing #77

Open abhigoku10 opened 3 years ago

abhigoku10 commented 3 years ago

@WeijingShi THanks for sharing the code base and its really nice work , we can use this work as baseline . I have few queries

Training

  1. Do we have multiclass model/ config where we have trained car,peds , bus,trucks class together and obtain overall predictions
  2. Kitti max range is 79m if we have to dataset with longer ranges what modifications have to made to train on data with long range>79m since the point cloud density of objects will be less
  3. To train on custom data what modification in the training pipeline have to be made since we would not have calibration files for custom data
  4. Can we train the model with distance as one of the parameter instead of the reflectance intensity of the lidar Testing :

  5. As mentioned in the paper the inference time of a point cloud : Reading the dataset and running the calibration takes 11.0% time (70ms). Creating the graph representation consumes 18.9% time (121ms). The inference of the GNN takes 56.4% time (363ms). Boxmerging and scoring take 13.1% time (84ms) so what steps have to been taken to optimize the inference for real time latency

    Futurescope: 1.Can we convert this PointGNN ot perform semanttic segmentation on point cloud data

  6. Can we train with nuscene , waymo and lyft dataset and check its accuracy with the current architecture

Thanks in advance

WeijingShi commented 3 years ago

Hi @abhigoku10, Good questions.

  1. We do not have pretrain models for more classes. If you are interested in training with car, pedestrian and cyclist together. You can modify the label_methodin the config files to "yaw", which should do the trick. Note that you need to choose a downsample density and graph radius that work for all classes.
    https://github.com/WeijingShi/Point-GNN/blob/48f3d79d5b101d3a4b8439ba74c92fcad4f7cab0/train.py#L108 https://github.com/WeijingShi/Point-GNN/blob/48f3d79d5b101d3a4b8439ba74c92fcad4f7cab0/dataset/kitti_dataset.py#L1132

As for bus, truck and others, Kitti does not contains enough samples. Nuscenes and Waymo Open are more suitable.

  1. The code does not put limits on the range. But this algorithm is based on lidar only. So if it can not get any object geometries due to the distance/sparsity, I don't think it can recognize well. Temporal aggregation or fusion can be good options.

  2. People had some successful on custom data: https://github.com/WeijingShi/Point-GNN/issues/64#issuecomment-762714959 If you can not covert your format to Kitti, you need to create you own custom_dataset.py. As long as it has a method to get_points and get_label, the rest pipeline should be the same.

  3. Sure, it would be very interesting to see the results.

  4. The graph creation and nms are both on CPU with python. Moving them to GPU and C++ should help. For the GNN inference part, we need find a way to prune edges and vertices without hurting the accuracy.

  5. We had this discussion before https://github.com/WeijingShi/Point-GNN/issues/24#issuecomment-650635419. To run segmentation, we need to remove the localization headers and only use the classification header. To keep the computational cost low, we need a FPN/Unet structure.

  6. Unfortunately, I don't have plan to work this in the short term. I am happy to support if you want to work on the tests. I think the challenges are computational cost for their dense point cloud.

Thanks,

hgoyal194 commented 2 years ago
  1. Note that you need to choose a downsample density and graph radius that works for all classes. Could you please explain to me how to choose these parameters for the Kitti dataset and also for other custom dataset for multiclass Thanks