search

sshaoshuai / PointRCNN

PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud, CVPR 2019.
MIT License
1.71k stars 425 forks source link

readme

PointRCNN

PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

teaser

Code release for the paper PointRCNN:3D Object Proposal Generation and Detection from Point Cloud, CVPR 2019.

Authors: Shaoshuai Shi, Xiaogang Wang, Hongsheng Li.

[arXiv]  [Project Page] 

New: We have provided another implementation of PointRCNN for joint training with multi-class in a general 3D object detection toolbox [OpenPCDet].

Introduction

In this work, we propose the PointRCNN 3D object detector to directly generated accurate 3D box proposals from raw point cloud in a bottom-up manner, which are then refined in the canonical coordinate by the proposed bin-based 3D box regression loss. To the best of our knowledge, PointRCNN is the first two-stage 3D object detector for 3D object detection by using only the raw point cloud as input. PointRCNN is evaluated on the KITTI dataset and achieves state-of-the-art performance on the KITTI 3D object detection leaderboard among all published works at the time of submission.

For more details of PointRCNN, please refer to our paper or project page.

Supported features and ToDo list

Installation

Requirements

All the codes are tested in the following environment:

Install PointRCNN

a. Clone the PointRCNN repository.

git clone --recursive https://github.com/sshaoshuai/PointRCNN.git

If you forget to add the --recursive parameter, just run the following command to clone the Pointnet2.PyTorch submodule.

git submodule update --init --recursive

b. Install the dependent python libraries like easydict,tqdm, tensorboardX etc.

c. Build and install the pointnet2_lib, iou3d, roipool3d libraries by executing the following command:

sh build_and_install.sh

Dataset preparation

Please download the official KITTI 3D object detection dataset and organize the downloaded files as follows: 

PointRCNN
├── data
│   ├── KITTI
│   │   ├── ImageSets
│   │   ├── object
│   │   │   ├──training
│   │   │      ├──calib & velodyne & label_2 & image_2 & (optional: planes)
│   │   │   ├──testing
│   │   │      ├──calib & velodyne & image_2
├── lib
├── pointnet2_lib
├── tools

Here the images are only used for visualization and the road planes are optional for data augmentation in the training.

Pretrained model

You could download the pretrained model(Car) of PointRCNN from here(~15MB), which is trained on the train split (3712 samples) and evaluated on the val split (3769 samples) and test split (7518 samples). The performance on validation set is as follows:

Car AP@0.70, 0.70, 0.70:
bbox AP:96.91, 89.53, 88.74
bev  AP:90.21, 87.89, 85.51
3d   AP:89.19, 78.85, 77.91
aos  AP:96.90, 89.41, 88.54

Quick demo

You could run the following command to evaluate the pretrained model (set RPN.LOC_XZ_FINE=False since it is a little different with the default configuration): 

python eval_rcnn.py --cfg_file cfgs/default.yaml --ckpt PointRCNN.pth --batch_size 1 --eval_mode rcnn --set RPN.LOC_XZ_FINE False

Inference

Here you could specify a bigger --batch_size for faster inference based on your GPU memory. Note that the --eval_mode argument should be consistent with the --train_mode used in the training process. If you are using --eval_mode=rcnn_offline, then you should use --rcnn_eval_roi_dir and --rcnn_eval_feature_dir to specify the saved features and proposals of the validation set. Please refer to the training section for more details.

Training

Currently, the two stages of PointRCNN are trained separately. Firstly, to use the ground truth sampling data augmentation for training, we should generate the ground truth database as follows:

python generate_gt_database.py --class_name 'Car' --split train

Training of RPN stage

After training, the checkpoints and training logs will be saved to the corresponding directory according to the name of your configuration file. Such as for the default.yaml, you could find the checkpoints and logs in the following directory:

PointRCNN/output/rpn/default/

which will be used for the training of RCNN stage.

Training of RCNN stage

Suppose you have a well-trained RPN model saved at output/rpn/default/ckpt/checkpoint_epoch_200.pth, then there are two strategies to train the second stage of PointRCNN.

(a) Train RCNN network with fixed RPN network to use online GT augmentation: Use --rpn_ckpt to specify the path of a well-trained RPN model and run the command as follows:

python train_rcnn.py --cfg_file cfgs/default.yaml --batch_size 4 --train_mode rcnn --epochs 70  --ckpt_save_interval 2 --rpn_ckpt ../output/rpn/default/ckpt/checkpoint_epoch_200.pth

(b) Train RCNN network with offline GT augmentation:

  1. Generate the augmented offline scenes by running the following command: 
    python generate_aug_scene.py --class_name Car --split train --aug_times 4
  2. Save the RPN features and proposals by adding --save_rpn_feature:

All the codes supported mutiple GPUs, simply add the --mgpus argument as above. And you could also increase the --batch_size by using multiple GPUs for training.

Note:

Citation

If you find this work useful in your research, please consider cite:

@InProceedings{Shi_2019_CVPR,
    author = {Shi, Shaoshuai and Wang, Xiaogang and Li, Hongsheng},
    title = {PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}