LiDAR Diffusion Models [CVPR 2024]

[**Haoxi Ran**](https://hancyran.github.io/) · [**Vitor Guizilini**](https://scholar.google.com.br/citations?user=UH9tP6QAAAAJ&hl=en) · [**Yue Wang**](https://yuewang.xyz/)

:tada: News :tada:

• [Apr 14, 2024] Pretrained autoencoders and LiDMs for different tasks are released!
• [Apr 5, 2024] Our codebase and a detailed study of our autoencoder design along with the pretrained models is released!

Requirements

We provide an available conda environment named lidar_diffusion:

sh init/create_env.sh
conda activate lidar_diffusion

Evaluation Toolbox

Overview of evaluation metrics:

To standardize the evaluation of LiDAR generative models, we provide a self-contained and mostly CUDA-accelerated evaluation toolbox in the directory ./lidm/eval/. It implements and integrates various evaluation metrics, including:

• Perceptual metrics:
  • Fréchet Range Image Distance (FRID)
  • Fréchet Sparse Volume Distance (FSVD)
  • Fréchet Point-based Volume Distance (FPVD)
• Statistical metrics:
  • Minimum Matching Distance (MMD)
  • Jensen-Shannon Divergence (JSD)
• Statistical pairwise metrics (for reconstruction only):
  • Chamfer Distance (CD)
  • Earth Mover's Distance (EMD)

For more details about setup and usage, please refer to the Evaluation Toolbox README.

Model Zoo

To test different tasks below, please download the pretrained LiDM and its corresponding autoencoder:

Pretrained Autoencoders

64-beam (evaluated on KITTI-360 val):

Benchmark for Unconditional LiDAR Generation

64-beam (2k samples):

NOTE:

1. Each method is evaluated with 2,000 randomly generated samples.
2. †: samples generated by the officially released pretrained model in LiDARGen github repo.
3. §: samples borrowed from UltraLiDAR implementation.
4. All above results are calculated from our evaluation toolbox. For more details, please refer to Evaluation Toolbox README.
5. Each .pcd file is a list of point clouds stored by joblib package. To load those files, use command joblib.load(path).

To evaluate above methods (except LiDM) yourself, download our provided .pcd files in the Output column to directory ./models/baseline/kitti/[method]/:

CUDA_VISIBLE_DEVICES=0 python scripts/sample.py -d kitti -f models/baseline/kitti/[method]/samples.pcd --baseline --eval

To evaluate LiDM through the given .pcd files:

CUDA_VISIBLE_DEVICES=0 python scripts/sample.py -d kitti -f models/lidm/kitti/[method]/samples.pcd --eval

Pretrained LiDMs for Other Tasks

NOTE:

1. The output log.tar.gz contains input conditions (.png), generated range images (.png), generated point clouds (.txt), and a collection of all output point clouds (.pcd).

Study on Design of LiDAR Compression

For full details of our studies on the design of LiDAR Compression, please refer to LiDAR Compression Design README.

Tip: Download the video instead of watching it with the Google Drive's built-in video player provides a better visualization.

Autoencoders (trained with 40k steps, evaluated on reconstruction):

Unconditional LiDAR Generation

To run sampling on pretrained models (and to evaluate your results with flag "--eval"), firstly download our provided pretrained autoencoders to directory ./models/first_stage_models/kitti/[model_name] and pretrained LiDMs to directory ./models/lidm/kitti/[model_name]:

CUDA_VISIBLE_DEVICES=0 python scripts/sample.py -d kitti -r models/lidm/kitti/[model_name]/model.ckpt -n 2000 --eval

Semantic-Map-to-LiDAR

To check the conditional results on a full sequence of semantic maps (sequence '08'), please refer to this video

Before run this task, set up the SemanticKITTI dataset first for semantic labels as input.

To run sampling on pretrained models (and to evaluate your results with flag "--eval"):

CUDA_VISIBLE_DEVICES=0 python scripts/sample_cond.py -r models/lidm/kitti/sem2lidar/model.ckpt -d kitti [--eval]

Camera-to-LiDAR

Before run this task, set up the KITTI-360 dataset first for camera images as input.

To run sampling on pretrained models:

CUDA_VISIBLE_DEVICES=0 python scripts/sample_cond.py -r models/lidm/kitti/sem2lidar/model.ckpt -d kitti [--eval]

Text-to-LiDAR

To run sampling on pretrained models:

CUDA_VISIBLE_DEVICES=0 python scripts/text2lidar.py -r models/lidm/kitti/cam2lidar/model.ckpt -d kitti -p "an empty road with no object"

Training

Besides, to train your own LiDAR Diffusion Models, just run this command (for example, train both autoencoder and lidm on four gpus):

# train an autoencoder
python main.py -b configs/autoencoder/kitti/autoencoder_c2_p4.yaml -t --gpus 0,1,2,3

# train an LiDM
python main.py -b configs/lidar_diffusion/kitti/uncond_c2_p4.yaml -t --gpus 0,1,2,3

To debug the training process, just add flag -d:

python main.py -b path/to/your/config.yaml -t --gpus 0, -d

To resume your training from an existing log directory or an existing checkpoint file, use the flag -r:

# using a log directory
python main.py -b path/to/your/config.yaml -t --gpus 0, -r path/to/your/log

# or, using a checkpoint 
python main.py -b path/to/your/config.yaml -t --gpus 0, -r path/to/your/ckpt/file

Acknowledgement

• Our codebase for the diffusion models builds heavily on Latent Diffusion

Citation

If you find this project useful in your research, please consider citing:

@inproceedings{ran2024towards,
    title={Towards Realistic Scene Generation with LiDAR Diffusion Models},
    author={Ran, Haoxi and Guizilini, Vitor and Wang, Yue},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024}
}