TL;DR: Our paper delve into enhancing SSC through the utilization of instance-centric representations. We propose a novel paradigm that integrates instance queries to facilitate instance semantics and capture global context. Our approach achieves SOTA results of 15.04 & 18.58 mIoU on the SemanticKITTI & KITTI-360, respectively.
This project is built upon TmPL, a template for rapid & flexible DL experimentation development built upon Lightning & Hydra.
Install the rest of the requirements with pip.
pip install -r requirements.txt
SemanticKITTI: Download the RGB images, calibration files, and preprocess the labels, referring to the documentation of VoxFormer or MonoScene.
SSCBench-KITTI-360: Refer to https://github.com/ai4ce/SSCBench/tree/main/dataset/KITTI-360.
SemanticKITTI: Generate depth predications with pre-trained MobileStereoNet referring to VoxFormer https://github.com/NVlabs/VoxFormer/tree/main/preprocess#3-image-to-depth.
SSCBench-KITTI-360: Follow the same procedure as SemanticKITTI but ensure to adapt the disparity value.
The pretrained weight of MaskDINO can be downloaded here.
Setup
export PYTHONPATH=`pwd`:$PYTHONPATH
Training
python tools/train.py [--config-name config[.yaml]] [trainer.devices=4] \
[+data_root=$DATA_ROOT] [+label_root=$LABEL_ROOT] [+depth_root=$DEPTH_ROOT]
--config-name
.Testing
Generate the outputs for submission on the evaluation server:
python tools/test.py [+ckpt_path=...]
Visualization
Generating outputs
python tools/generate_outputs.py [+ckpt_path=...]
Visualization
python tools/visualize.py [+path=...]
SemanticKITTI
Method | Split | IoU | mIoU | Download |
---|---|---|---|---|
Symphonies | val | 41.92 | 14.89 | log / model |
Symphonies | test | 42.19 | 15.04 | output |
KITTI-360
Method | Split | IoU | mIoU | Download |
---|---|---|---|---|
Symphonies | test | 44.12 | 18.58 | log / model |
If you find our paper and code useful for your research, please consider giving this repo a star :star: or citing :pencil::
@article{jiang2023symphonies,
title={Symphonize 3D Semantic Scene Completion with Contextual Instance Queries},
author={Haoyi Jiang and Tianheng Cheng and Naiyu Gao and Haoyang Zhang and Tianwei Lin and Wenyu Liu and Xinggang Wang},
journal={CVPR},
year={2024}
}
The development of this project is inspired and informed by MonoScene, MaskDINO and VoxFormer. We are thankful to build upon the pioneering work of these projects.
Released under the MIT License.