search

darrenjkt / MS3D

Auto-labeling of point cloud sequences for 3D object detection using an ensemble of experts and temporal refinement
Apache License 2.0
141 stars 16 forks source link

Auto-labelling new classes #28

Open jamesheatonrdm opened 1 month ago

jamesheatonrdm commented 1 month ago

I am looking to auto-label a dataset that contains objects similar to those found nuScenes, specifially the trailer class.

As far as I understand it, this work only currently supports auto-labelling of pedestrians and vehicles.

Currently, I am wondering if the models trained on nuScenes data have been trained on all of the classes in this dataset, or just [Car, Truck, Bus, Motorcycle, Pedestrian]. If I look in the config files e.g. this one, there are just the five classes there.

I have two questions:

  1. Is it possible to use the pretrained nuScenes models for auto-labelling different classes or do I have to retrain from scratch?
  2. If I am retraining from scratch, where can I find the parameters that you used to pre-train, so that I can keep everything the same except for the number of classes.

Thanks in advance

darrenjkt commented 1 month ago

The current pre-trained nuscenes models have only been trained on [Car, Truck, Bus, Motorcycle, Pedestrian]. You'll have to retrain the nuscenes models to detect the trailer class.

  1. If you want to use pre-trained models to detect classes it has not seen in training, you should look at papers in the field of object discovery or open-world object detection.
  2. To train from scratch, you can follow the config file that you linked. The only thing is to make sure to set the number of sweeps and adapt the classes. The specific config used to train our nuscenes voxelRCNN (anchorhead) is this file.

Adding new classes When adding new classes to anchor-based detectors, you need to make sure to copy over the anchor-size. Use OpenPCDet's SECOND for reference for nuScenes/waymo/kitti/etc. Each dataset has a different default anchor-size cause they use the average size of the object class in that dataset.

https://github.com/open-mmlab/OpenPCDet/blob/8cacccec11db6f59bf6934600c9a175dae254806/tools/cfgs/nuscenes_models/cbgs_second_multihead.yaml#L81-L90

For centerhead, make sure you modify the class names for the detector head.

https://github.com/open-mmlab/OpenPCDet/blob/8cacccec11db6f59bf6934600c9a175dae254806/tools/cfgs/nuscenes_models/cbgs_dyn_pp_centerpoint.yaml#L47-L54

And last but not least, don't forget to change the CLASS_NAMES for both anchor and center head. https://github.com/darrenjkt/MS3D/blob/a35d6cc8ef8cc55e68a5a9e260917eac336b599a/tools/cfgs/nuscenes_models/uda_voxel_rcnn_anchorhead.yaml#L1

jamesheatonrdm commented 1 month ago

@darrenjkt Thanks for the detailed response! You have been very helpful

jamesheatonrdm commented 1 month ago

Hi @darrenjkt

Sorry to come back to this but I have run into an issue with one of the configs.

I have managed to add the additional class and train the models successfully except for IA-SSD (I did have to add PRED_VELOCITY: False in the DATA_CONFIG for each, I am presuming this is fine).

With the IA-SSD config, I get the following error in OpenPCDet data_processor.py:

AttributeError: 'DataProcessor' object has no attribute 'sample_points_by_voxels'

I have looked for the same issue on the OpenPCDet page but have unfortunately not found anyone with the same issue.

Do you know what could be causing this and how to correct the issue?

Thanks again.

darrenjkt commented 1 month ago

For IA-SSD, are you training it in this MS3D repo? It won't work on OpenPCDet because they don't support it.

I got the code from the original IA-SSD repo: https://github.com/yifanzhang713/IA-SSD