ldkong1205/LaserMix - Githubissues

LaserMix for Semi-Supervised LiDAR Semantic Segmentation

Lingdong Kong, Jiawei Ren, Liang Pan, Ziwei Liu
S-Lab, Nanyang Technological University

About

LaserMix is a semi-supervised learning (SSL) framework designed for LiDAR semantic segmentation. It leverages the strong spatial prior of driving scenes to construct low-variation areas via laser beam mixing, and encourages segmentation models to make confident and consistent predictions before and after mixing.

Fig. Illustration for laser beam partition based on inclination φ.

Visit our project page to explore more details. :red_car:

Updates

[2024.05] - Our improved framework, LaserMix++ :rocket:, is avaliable on arXiv.
[2024.01] - The toolkit tailored for The RoboDrive Challenge has been released. :hammer_and_wrench:
[2023.12] - We are hosting The RoboDrive Challenge at ICRA 2024. :blue_car:
[2023.12] - Introducing FRNet, an efficient and effective real-time LiDAR segmentation model that achieves promising semi-supervised learning results on SemanticKITTI and nuScenes. Code and checkpoints are available for downloading.
[2023.03] - Intend to test the robustness of your LiDAR semantic segmentation models? Check our recent work, :robot: Robo3D, a comprehensive suite that enables OoD robustness evaluation of 3D segmentors on our newly established datasets: SemanticKITTI-C, nuScenes-C, and WOD-C.
[2023.03] - LaserMix was selected as a :sparkles: highlight :sparkles: at CVPR 2023 (top 10% of accepted papers).
[2023.02] - LaserMix was accepted to CVPR 2023! :tada:
[2023.02] - LaserMix has been integrated into the MMDetection3D codebase! Check this PR in the dev-1.x branch to know more details. :beers:
[2023.01] - As suggested, we will establish a sequential track taking into account the LiDAR data collection nature in our semi-supervised LiDAR semantic segmentation benchmark. The results will be gradually updated in RESULT.md.
[2022.12] - We support a wider range of LiDAR segmentation backbones, including RangeNet++, SalsaNext, FIDNet, CENet, MinkowskiUNet, Cylinder3D, and SPVCNN, under both fully- and semi-supervised settings. The checkpoints will be available soon!
[2022.12] - The derivation of spatial-prior-based SSL is available here. Take a look! :memo:
[2022.08] - LaserMix achieves 1st place among the semi-supervised semantic segmentation leaderboards of nuScenes, SemanticKITTI, and ScribbleKITTI, based on Paper-with-Code. :bar_chart:
[2022.08] - We provide a video demo for visual comparisons on the SemanticKITTI val set. Take a look!
[2022.07] - Our paper is available on arXiv, click here to check it out. Code will be available soon!

Installation

Please refer to INSTALL.md for the installation details.

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ¹nuScenes, ²SemanticKITTI, and ³ScribbleKITTI datasets.

Getting Started

Please refer to GET_STARTED.md to learn more usage about this codebase.

Video Demo

Demo 1	Demo 2	Demo 3

Link ^{:arrow_heading_up:}	Link ^{:arrow_heading_up:}	Link ^{:arrow_heading_up:}

Main Result

Framework Overview

Range View

Method	nuScenes				SemanticKITTI				ScribbleKITTI
Method	1%	10%	20%	50%	1%	10%	20%	50%	1%	10%	20%	50%
Sup.-only	38.3	57.5	62.7	67.6	36.2	52.2	55.9	57.2	33.1	47.7	49.9	52.5
LaserMix	49.5	68.2	70.6	73.0	43.4	58.8	59.4	61.4	38.3	54.4	55.6	58.7
improv. ↑	⁺11.2	⁺10.7	⁺7.9	⁺5.4	⁺7.2	⁺6.6	⁺3.5	⁺4.2	⁺5.2	⁺6.7	⁺5.7	⁺6.2
LaserMix++
improv. ↑

Voxel

Method	nuScenes				SemanticKITTI				ScribbleKITTI
Method	1%	10%	20%	50%	1%	10%	20%	50%	1%	10%	20%	50%
Sup.-only	50.9	65.9	66.6	71.2	45.4	56.1	57.8	58.7	39.2	48.0	52.1	53.8
LaserMix	55.3	69.9	71.8	73.2	50.6	60.0	61.9	62.3	44.2	53.7	55.1	56.8
improv. ↑	⁺4.4	⁺4.0	⁺5.2	⁺2.0	⁺5.2	⁺3.9	⁺4.1	⁺3.6	⁺5.0	⁺5.7	⁺3.0	⁺3.0
LaserMix++
improv. ↑

Ablation Studies

Qualitative Examples

qualitative

Checkpoints & More Results

For more experimental results and pretrained weights, please refer to RESULT.md.

TODO List

[x] Initial release. :rocket:
[x] Add license. See here for more details.
[x] Add video demos :movie_camera:
[x] Add installation details.
[x] Add data preparation details.
[ ] Add evaluation details.
[ ] Add training details.

Citation

If you find this work helpful, please kindly consider citing our paper:

@inproceedings{kong2023lasermix,
  title = {LaserMix for Semi-Supervised LiDAR Semantic Segmentation},
  author = {Kong, Lingdong and Ren, Jiawei and Pan, Liang and Liu, Ziwei},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages = {21705--21715},
  year = {2023},
}

License

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

This work is developed based on the MMDetection3D codebase.

MMDetection3D is an open-source toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

We acknowledge the use of the following public resources during the course of this work: ¹nuScenes, ²nuScenes-devkit, ³SemanticKITTI, ⁴SemanticKITTI-API, ⁵ScribbleKITTI, ⁶FIDNet, ⁷CENet, ⁸SPVNAS, ⁹Cylinder3D, ¹⁰TorchSemiSeg, ¹¹MixUp, ¹²CutMix, ¹³CutMix-Seg, ¹⁴CBST, ¹⁵MeanTeacher, and ¹⁶Cityscapes.

We would like to thank Fangzhou Hong for the insightful discussions and feedback. ❤️