This repository is the official Pytorch implementation of our work:
[ICLR 2023] CRB: Exploring Active 3D Object Detection from a Generalization Perspective.
[OpenReview] [arXiv] [Supplementary Material]
[In Submission] Open-CRB: Towards Open World Active Learning for 3D Object Detection.
:fire: 11/23 updates: release the code and the preprint of Open-CRB
:fire: 02/23 updates: checkpoints available at https://drive.google.com/drive/folders/1PMb6tu84AIw66vCRrMBCHpnBeL5WMkuv?usp=sharing
To alleviate the high annotation cost in LiDAR-based 3D object detection, active learning is a promising solution that learns to select only a small portion of unlabeled data to annotate, without compromising model performance. Our empirical study, however, suggests that mainstream uncertainty-based and diversity-based active learning policies are not effective when applied in the 3D detection task, as they fail to balance the trade-off between point cloud informativeness and box-level annotation costs. To overcome this limitation, we jointly investigate three novel criteria in our framework CRB for point cloud acquisition - label conciseness, feature representativeness and geometric balance, which hierarchically filters out the point clouds of redundant 3D bounding box labels, latent features and geometric characteristics (e.g., point cloud density) from the unlabeled sample pool and greedily selects informative ones with fewer objects to annotate. Our theoretical analysis demonstrates that the proposed criteria aligns the marginal distributions of the selected subset and the prior distributions of the unseen test set, and minimizes the upper bound of the generalization error. To validate the effectiveness and applicability of CRB, we conduct extensive experiments on the two benchmark 3D object detection datasets of KITTI and Waymo and examine both one-stage (i.e., SECOND) and two-stage 3D detectors (i.e., PV-RCNN). Experiments evidence that the proposed approach outperforms existing active learning strategies and achieves fully supervised performance requiring 1\% and 8\% annotations of bounding boxes and point clouds, respectively.
All the codes are tested in the following environment:
spconv-cu113 v2.1.21
pcdet v0.5
Our implementations of 3D detectors are based on the lastest OpenPCDet
. To install this pcdet
library and its dependent libraries, please run the following command:
python setup.py develop
NOTE: Please re-install even if you have already installed pcdet previoursly.
The active learning configs are located at tools/cfgs/active-kitti_models and /tools/cfgs/active-waymo_models for different AL methods. The dataset configs are located within tools/cfgs/dataset_configs, and the model configs are located within tools/cfgs for different datasets.
Currently we provide the dataloader of KITTI dataset and Waymo dataset, and the supporting of more datasets are on the way.
pcdet v0.1
, you can choose to use the old infos and set the DATABASE_WITH_FAKELIDAR option in tools/cfgs/dataset_configs/kitti_dataset.yaml as True. The second choice is that you can create the infos and gt database again and leave the config unchanged.CRB-active-3Ddet
├── data
│ ├── kitti
│ │ │── ImageSets
│ │ │── training
│ │ │ ├──calib & velodyne & label_2 & image_2 & (optional: planes) & (optional: depth_2)
│ │ │── testing
│ │ │ ├──calib & velodyne & image_2
├── pcdet
├── tools
python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml
Please download the official Waymo Open Dataset,
including the training data training_0000.tar~training_0031.tar
and the validation
data validation_0000.tar~validation_0007.tar
.
Unzip all the above xxxx.tar
files to the directory of data/waymo/raw_data
as follows (You could get 798 train tfrecord and 202 val tfrecord ):
CRB-active-3Ddet
├── data
│ ├── waymo
│ │ │── ImageSets
│ │ │── raw_data
│ │ │ │── segment-xxxxxxxx.tfrecord
| | | |── ...
| | |── waymo_processed_data_v0_5_0
│ │ │ │── segment-xxxxxxxx/
| | | |── ...
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1/
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy (optional)
│ │ │── waymo_processed_data_v0_5_0_infos_train.pkl (optional)
│ │ │── waymo_processed_data_v0_5_0_infos_val.pkl (optional)
├── pcdet
├── tools
Install the official waymo-open-dataset
by running the following command:
pip3 install --upgrade pip
pip3 install waymo-open-dataset-tf-2-0-0==1.2.0 --user
Waymo version in our project is 1.2.0
Extract point cloud data from tfrecord and generate data infos by running the following command (it takes several hours,
and you could refer to data/waymo/waymo_processed_data_v0_5_0
to see how many records that have been processed):
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
--cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml
Note that you do not need to install waymo-open-dataset
if you have already processed the data before and do not need to evaluate with official Waymo Metrics.
The weights of our pre-trained model will be released upon acceptance.
Test with a pretrained model:
python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}
To test all the saved checkpoints of a specific training setting and draw the performance curve on the Tensorboard, add the --eval_all
argument:
python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --eval_all
To test with multiple GPUs:
sh scripts/dist_test.sh ${NUM_GPUS} \
--cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_GPUS} \ --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
### Train a backbone
In our active learning setting, the 3D detector will be pre-trained with a small labeled set $\mathcal{D}_L$ which is randomly sampled from the trainig set. To train such a backbone, please run
```shell script
sh scripts/${DATASET}/train_${DATASET}_backbone.sh
We provide several options for active learning algorithms, including
random
]confidence
]entropy
]montecarlo
]coreset
]llal
]badge
]crb
]You could optionally add extra command line parameters --batch_size ${BATCH_SIZE}
and --epochs ${EPOCHS}
to specify your preferred parameters.
python train.py --cfg_file ${CONFIG_FILE}