search

CollaborativePerception / V2Xverse

90 stars 10 forks source link

readme

Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System (Beta)

Paper | Project page

This repository contains the official PyTorch implementation of paper "Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System".

V2X autonomous driving

Features

Support the developing of our CoDriving system in three tasks:

Support the deployments of SOTA end-to-end autonomous driving methods in Carla-based benchmark:

Support the complete developing pipeline (training + offline evaluation + closed-loop driving evaluation) of multiple collaborative perception methods:

Modality:

Contents

  1. Installation
  2. Dataset
  3. Training
  4. Closed loop evaluation
  5. Modular evaluation
  6. Shutdown simulation
  7. Todo
  8. Acknowledgements

Installation

Step 1: Basic Installation

Get code and create pytorch environment.

git clone https://github.com/CollaborativePerception/V2Xverse.git
conda create --name v2xverse python=3.7 cmake=3.22.1
conda activate v2xverse
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
conda install cudnn -c conda-forge

cd V2Xverse
pip install -r opencood/requirements.txt
pip install -r simulation/requirements.txt

Step 2: Download and setup CARLA 0.9.10.1.

chmod +x simulation/setup_carla.sh
./simulation/setup_carla.sh
easy_install carla/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg
mkdir external_paths
ln -s ${PWD}/carla/ external_paths/carla_root
# If you already have a Carla, just create a soft link to external_paths/carla_root

Note: we choose the setuptools==41 to install because this version has the feature easy_install. After installing the carla.egg you can install the lastest setuptools to avoid No module named distutils_hack.

Step 3: Install Spconv (1.2.1)

We use spconv 1.2.1 to generate voxel features in perception module.

To install spconv 1.2.1, please follow the guide in https://github.com/traveller59/spconv/tree/v1.2.1.

Step 4: Set up opencood

# Set up
python setup.py develop

# Bbx IOU cuda version compile
python opencood/utils/setup.py build_ext --inplace

Step 5: Install pypcd

# go to another folder
cd ..
git clone https://github.com/klintan/pypcd.git
cd pypcd
pip install python-lzf
python setup.py install
cd ..

Step 6: Install EfficinetNet(required by camera detector Lift-Splat-Shoot)

pip install efficientnet_pytorch==0.7.0

Dataset

There are two ways to obtain dataset, you can generate a dataset by youself or download one from google drivie(coming soon). Here are the steps to generate a dataset, where we employ a strong privileged rule-based expert agent as supervisor.

# Generate a dataset in parallel

cd V2Xverse
# Initialize dataset directory
python ./simulation/data_collection/init_dir.py --dataset_dir  ./dataset

# Generate scripts for every routes
python ./simulation/data_collection/generate_scripts.py

# Link dataset directory, if you initialized dataset in other directory, replace ./dataset with your dataset directory
ln -s ${PWD}/dataset/ ./external_paths/data_root

# Open Carla server (15 parallel process in total)
CUDA_VISIBLE_DEVICES=0 ./external_paths/carla_root/CarlaUE4.sh --world-port=40000 -prefer-nvidia
CUDA_VISIBLE_DEVICES=1 ./external_paths/carla_root/CarlaUE4.sh --world-port=40002 -prefer-nvidia
CUDA_VISIBLE_DEVICES=2 ./external_paths/carla_root/CarlaUE4.sh --world-port=40004 -prefer-nvidia
...
CUDA_VISIBLE_DEVICES=7 ./external_paths/carla_root/CarlaUE4.sh --world-port=40028 -prefer-nvidia

# Execute data generation in parallel
bash simulation/data_collection/generate_v2xverse_all.sh

Generate data on one single route.

# Open one Carla server
CUDA_VISIBLE_DEVICES=0 ./external_paths/carla_root/CarlaUE4.sh --world-port=40000 -prefer-nvidia

# Execute data generation for route 0 in town01
bash ./simulation/data_collection/scripts/weather-0/routes_town01_0.sh

Tips: set usable --world-port and adjust ${PORT} in /simulation/data_collection/scripts/weather-0/routes_townXX_X.sh accordingly. Otherwise, the python programme might stuck.

The files in dataset should follow this structure:

|--weather-0
    |--data
        |--routes_town{town_id}_{route_id}_w{weather_id}_{datetime}
            |--ego_vehicle_{vehicle_id}
                |--2d_bbs_{direction}
                |--3d_bbs
                |--actors_data
                |--affordances
                |--bev_visibility
                |--birdview
                |--depth_{direction}
                |--env_actors_data
                |--lidar
                |--lidar_semantic_front
                |--measurements
                |--rgb_{direction}
                |--seg_{direction}
                |--topdown
            |--rsu_{vehicle_id}
            |--log
    |--results
...
|--weather-13

Once a new dataset is generated in ./dataset, generate a index file with:

python simulation/data_collection/gen_index.py

This will result in dataset/dataset_index.txt, from which we retrieval dataset sub-directory in training and testing.

Training

Perception module

We use yaml files to configure parameters to train perception module. See opencood/hypes_yaml/v2xverse/ for examples.

To train perception module from scratch or a continued checkpoint, run the following commonds:

python opencood/tools/train.py -y ${CONFIG_FILE} [--model_dir ${CHECKPOINT_FOLDER}]

Arguments Explanation:

Train the perception module in DDP:

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch  --nproc_per_node=2 --use_env opencood/tools/train_ddp.py -y ${CONFIG_FILE} [--model_dir ${CHECKPOINT_FOLDER}]

--nproc_per_node indicate the GPU number you will use.

Test the perception module:

python opencood/tools/inference_multiclass.py --model_dir ${CHECKPOINT_FOLDER}

Test the perception module in latency setting:

python opencood/tools/inference_multiclass_latency.py --model_dir ${CHECKPOINT_FOLDER}

Test the perception module in pose error setting:

python opencood/tools/inference_multiclass_w_noise.py --model_dir ${CHECKPOINT_FOLDER}

Planning module

Given a checkpoint of perception module, we freeze its parameters and train the down-stream planning module (MotionNet as backbone) in an end-to-end paradigm. The planner gets BEV perception feature and occupancy map as input and targets to predict the future waypoints of ego vehicle.

Train the planning module with a given perception checkpoint:

bash scripts/train_planner_e2e.sh ${CUDA_VISIBLE_DEVICES} ${NUM_GPUS} ${perception_model_dir} ${collaboration_method} ${planner_resume}

Arguments Explanation:

Test the entire driving system (perception+planning) in waypoints prediction task with ADE and FDE:

bash scripts/eval_planner_e2e.sh  ${CUDA_VISIBLE_DEVICES} ${perception_model_dir} ${collaboration_method} ${planner_resume}

This evaluation measures the ability of driving system to clone the behaviors of expert agent.

Test the waypoints prediction task in latency setting:

bash scripts/eval_planner_e2e_latency.sh  ${CUDA_VISIBLE_DEVICES} ${perception_model_dir} ${collaboration_method} ${planner_resume}

Test the waypoints prediction task in pose error setting:

bash scripts/eval_planner_e2e_w_noise.sh  ${CUDA_VISIBLE_DEVICES} ${perception_model_dir} ${collaboration_method} ${planner_resume}

Closed-loop evaluation

Your can customize closed-loop evaluation with specific agents and scenarios. For evaluation on one route, following these steps:

# Open one Carla server
CUDA_VISIBLE_DEVICES=0 ./external_paths/carla_root/CarlaUE4.sh --world-port=${Carla_port} -prefer-nvidia

# Evaluation on one route
CUDA_VISIBLE_DEVICES=0 bash scripts/eval_driving_e2e.sh ${Route_id} ${Carla_port} ${Method_tag} ${Repeat_id} ${Agent_config} ${Scenario_config}

Arguments Explanation:

Shut down simulation on Linux

Carla processes may fail to stop，please kill them in time.

Display your processes

ps U usrname | grep PROCESS_NAME(eg. python，carla)

Kill process

kill -9 PID

Kill all carla-related processes

ps -def |grep 'carla' |cut -c 9-15| xargs kill -9
pkill -u username -f carla

Todo

Acknowledgements

This implementation is based on code from several repositories.

Citation

@article{liu2024codriving,
  title={Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System},
  author={Liu, Genjia and Hu, Yue and Xu, Chenxin and Mao, Weibo and Ge, Junhao and Huang, Zhengxiang and Lu, Yifan and Xu, Yinda and Xia, Junkai and Wang, Yafei and others},
  journal={arXiv preprint arXiv:2404.09496},
  year={2024}
}