search

byzhaoAI / BM2CP

Other
14 stars 3 forks source link

readme

BM2CP

[CoRL 2023] BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities

Overview

Where2comm Abstract: Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios.

Updates

Features

Quick Start

Download dataset

1. OPV2V

Download raw data of OPV2V relseased by Official.

2. DAIR-V2X
  1. Download raw data of DAIR-V2X.
  2. Download complemented annotation from Yifan Lu.

Install

Please refer to the INSTALL.md for detailed documentations.

Train your model

First of all, modify the dataset path in the setting file, i.e. xxx.yaml.

data_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure"
root_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/train.json"
validate_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/val.json"
test_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/val.json"

The setting is same as OpenCOOD, which uses yaml file to configure all the parameters for training. To train your own model from scratch or a continued checkpoint, run the following commonds:

python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER}]

Arguments Explanation:

For example, to train BM2CP from scratch:

python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/dair-v2x/dair_bm2cp.yaml

To train BM2CP from a checkpoint:

python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/dair-v2x/dair_bm2cp.yaml --model_dir opencood/logs/dair_bm2cp_2023_11_28_08_52_46

Test the model

Before you run the following command, first make sure the validation_dir in config.yaml under your checkpoint folder refers to the testing dataset path, e.g. opv2v_data_dumping/test.

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} --eval_epoch ${epoch_number} --save_vis ${default False}

Arguments Explanation:

The evaluation results will be dumped in the model directory.

Citation

If you are using our project for your research, please cite the following paper:


@InProceedings{zhao2023bm,
  title = {BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities},
  author = {Zhao, Binyu and ZHANG, Wei and Zou, Zhaonian},
  booktitle = {Proceedings of The 7th Conference on Robot Learning},
  pages = {1022--1035},
  year = {2023},
  series = {Proceedings of Machine Learning Research},
}

Acknowledgements

Thank for the excellent cooperative perception codebases OpenCOOD, CoPerception and Where2comm.

Thank for the excellent cooperative perception datasets DAIR-V2X, OPV2V and V2X-SIM.

Thank for the dataset and code support by DerrickXu, Yue Hu and YiFan Lu.

Relevant Projects

Thanks for the insightful previous works in cooperative perception field.

Methods

V2VNet: Vehicle-to-vehicle communication for joint perception and prediction ECCV20 [Paper]

When2com: Multi-agent perception via communication graph grouping CVPR20 [Paper] [Code]

Learning Distilled Collaboration Graph for Multi-Agent Perception NeurIPS21 [Paper] [Code]

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer ECCV2022 [Paper] [Code] [Talk]

Self-Supervised Collaborative Scene Completion: Towards Task-Agnostic Multi-Robot Perception CoRL2022 [Paper]

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers CoRL2022 [Paper] [Code]

Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps NeurIPS2022 [Paper] [Code]

Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception ICCV2023 [Paper][Code]

How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perceptio NeurIPS2023 [Paper] [Code]

Datasets

OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication ICRA2022 [Paper] [Website] [Code]

V2X-Sim: A Virtual Collaborative Perception Dataset and Benchmark for Autonomous Driving RAL21 [Paper] [Website][Code]

DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection CVPR2022 [Paper] [Website] [Code]