LRRU: Long-short Range Recurrent Updating Networks for Depth Completion (ICCV 2023)

Project Page, arXiv

Environment

CUDA 12.0
CUDNN 8.5.0
torch 1.7.1
torchvision 0.8.0
pip install -r LRRU/requirements.txt
pip3 install opencv-python
pip3 install opencv-python-headless

Wandb

We used WANDB to visualize and track our experiments.

NVIDIA Apex

We used NVIDIA Apex for multi-GPU training as NLSPN.

Apex can be installed as follows:

$ cd PATH_TO_INSTALL
$ git clone https://github.com/NVIDIA/apex
$ cd apex
$ pip install -v --disable-pip-version-check --no-build-isolation --no-cache-dir ./

Deformable Convolution V2 (DCNv2)

Build and install DCN module following here.

Dataset

KITTI Depth Completion (KITTI DC)

KITTI DC dataset is available at the KITTI DC Website and the data structure is:

.
├── depth_selection
│    ├── test_depth_completion_anonymous
│    │    ├── image
│    │    ├── intrinsics
│    │    └── velodyne_raw
│    ├── test_depth_prediction_anonymous
│    │    ├── image
│    │    └── intrinsics
│    └── val_selection_cropped
│        ├── groundtruth_depth
│        ├── image
│        ├── intrinsics
│        └── velodyne_raw
├── train
│    ├── 2011_09_26_drive_0001_sync
│    │    ├── image_02
│    │    │     └── data
│    │    ├── image_03
│    │    │     └── data
│    │    ├── oxts
│    │    │     └── data
│    │    └── proj_depth
│    │        ├── groundtruth
│    │        └── velodyne_raw
│    └── ...
└── val
    ├── 2011_09_26_drive_0002_sync
    └── ...

Usage

Training

$ sh train.sh

# train LRRU_Mini model
# python LRRU/train_apex.py -c train_lrru_mini_kitti.yml

# train LRRU_Tiny model
# python LRRU/train_apex.py -c train_lrru_tiny_kitti.yml

# train LRRU_Small model
# python LRRU/train_apex.py -c train_lrru_small_kitti.yml

# train LRRU_Base model
# python LRRU/train_apex.py -c train_lrru_base_kitti.yml

Testing

# download the pretrained model and add it to corresponding path.

$ sh val.sh

# val LRRU_Mini model
# python LRRU/val.py -c val_lrru_mini_kitti.yml

# val LRRU_Tiny model
# python LRRU/val.py -c val_lrru_tiny_kitti.yml

# val LRRU_Small model
# python LRRU/val.py -c val_lrru_small_kitti.yml

# val LRRU_Base model
# python LRRU/val.py -c val_lrru_base_kitti.yml

Pretrained models

Models on the KITTI validate dataset.

Acknowledgments

Thanks the ACs and the reviewers for their insightful comments, which are very helpful to improve our paper!

We are especially grateful to NLSPN, for their novel work and the excellent open source code! We are appreciative for IP_Basic, GuideNet, and DySPN, which have inspired us in model design.

In addition, thanks for all open source projects that have effectively promoted the development of the depth completion communities!

Non-learning methods: RAL_Non-Learning_DepthCompletion,

Supervised methods： S2D, CSPN, PENet, ACMNet, MDANet, DeepLiDAR, MSG-CHN, Sparse-Depth-Completion, GAENet,
ABCD, SemAttNet, CompletionFormer, ReDC.

Unsupervised methods： S2D, ScaffFusion-SSL, KBNet, ScaffFusion, VOICED.

If I have accidentally forgotten your work, please contact me to add.

Citation

@InProceedings{LRRU_ICCV_2023,
  author    = {Wang, Yufei and Li, Bo and Zhang, Ge and Liu, Qi and Gao Tao and Dai, Yuchao},
  title     = {LRRU: Long-short Range Recurrent Updating Networks for Depth Completion},
  booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
  year      = {2023},
}