[ECCV 2024 Oral] Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather

Junsung Park, Kyungmin Kim, Hyunjung Shim
CVML Lab. KAIST AI.


[Project Page]

About

Official implementation of "Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather", accepted in ECCV 2024.

Existing LiDAR semantic segmentation methods often struggle in adverse weather conditions. Previous work has addressed this by simulating adverse weather or using general data augmentation, but lacks detailed analysis of the negative effects on performance. We identified key factors of adverse weather affecting performance: geometric perturbation from refraction and point drop due to energy absorption and occlusions. Based on these findings, we propose new data augmentation techniques: Selective Jittering (SJ) to mimic geometric perturbation and Learnable Point Drop (LPD) to approximate point drop patterns using a Deep Q-Learning Network. These techniques enhance the model by exposing it to identified vulnerabilities without precise weather simulation.

Fig. The overall training process of our methods.

Updates

• [2024.09] - Our paper is selected as ORAL PRESENTATION in ECCV 2024! Link
• [2024.08] - Our project page is opened! Check it out in here!
• [2024.08] - Official implementation is released! Also, our paper is available on arXiv, click here to check it out.

Contents

• Installation
• Data Preparation
• Getting Started
• Main Results
• License
• Acknowledgement
• Citation

Installation

conda create -n lidar_weather python=3.8 -y && conda activate lidar_weather
conda install pytorch==1.10.0 torchvision==0.11.0 cudatoolkit=11.3 -c pytorch -y
pip install -U openmim && mim install mmengine && mim install 'mmcv>=2.0.0rc4, <2.1.0' && mim install 'mmdet>=3.0.0, <3.2.0'

git clone https://github.com/engineerJPark/LiDARWeather.git
cd LiDARWeather && pip install -v -e .

pip install cumm-cu113 && pip install spconv-cu113
sudo apt-get install libsparsehash-dev
export PATH=/usr/local/cuda/bin:$PATH && pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
pip install nuscenes-devkit
pip install wandb

Please refer to INSTALL.md for the installation details.

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ¹SemanticKITTI, ²SynLiDAR, ³SemanticSTF, and ⁴SemanticKITTI-C datasets.

Getting Started

Train

./tools/dist_train.sh configs/lidarweather_minkunet/sj+lpd+minkunet_semantickitti.py 4

./tools/dist_train.sh projects/CENet/lidarweather_cenet/sj+lpd+cenet_semantickitti.py 4

Test

python tools/test.py configs/lidarweather_minkunet/sj+lpd+minkunet_semantickitti.py work_dirs/sj+lpd+minkunet_semantickitti/epoch_15.pth

python tools/test.py projects/CENet/lidarweather_cenet/sj+lpd+cenet_semantickitti.py work_dirs/sj+lpd+cenet_semantickitti/epoch_50.pth

Please refer to GET_STARTED.md to learn more details.

Main Results

SemanticKITTI → SemanticSTF

SynLiDAR → SemanticSTF

Other Models \& Dataset

Qualitative Results

Fig. Qualitative results of our methods.

License

This work is under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Acknowledgement

Our codebase builds heavily on MMDetection3D and PyTorch DQN Tutorials. MMDetection3D is an open-source toolbox based on PyTorch, towards the next-generation platform for general 3D perception. It is a part of the OpenMMLab project developed by MMLab.

Citation

If you find this work helpful, please kindly consider citing our paper:

@article{park2024rethinking,
  title={Rethinking Data Augmentation for Robust LiDAR Semantic Segmentation in Adverse Weather},
  author={Park, Junsung and Kim, Kyungmin and Shim, Hyunjung},
  journal={arXiv preprint arXiv:2407.02286},
  year={2024}
}

This citation will be updated after the proceedings are published.