SfMNeXt: The NeXt Series of Learning Structure Prior from Motion

Official implementation of "SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimation" [AAAI 2024], and more.

Updates

2023.09.28 One paper accepted by AAAI 2024!

2023.09.03 Submitted the preprint of SQLdepth to Arxiv.

Upcoming releases

[x] release code for training in outdoor scenes (KITTI, Cityscapes)
[x] model release (KITTI, Cityscapes)
[x] code for training in indoor scenes (NYU-Depth-v2, MannequinChallenge)
[x] code for finetuning self-supervised model using metric depth
[ ] model release for indoor scenes and metric fine-tuned model

Training

To train on KITTI, run:

python train.py ./args_files/args_res50_kitti_192x640_train.txt

For instructions on downloading the KITTI dataset, see Monodepth2

To train on CityScapes, run:

python train.py ./args_files/args_cityscapes_train.txt

To finetune on CityScapes, run:

python train.py ./args_files/args_cityscapes_finetune.txt

For preparing cityscapes dataset, please refer to SfMLearner's prepare_train_data.py script. We used the following command:

python prepare_train_data.py \
    --img_height 512 \
    --img_width 1024 \
    --dataset_dir <path_to_downloaded_cityscapes_data> \
    --dataset_name cityscapes \
    --dump_root <your_preprocessed_cityscapes_path> \
    --seq_length 3 \
    --num_threads 8

💾 Pretrained weights and evaluation

You can download weights for some pretrained models here:

Methods	WxH	abs rel	RMSE
KITTI (ResNet-50)	640x192	0.088	4.175
KITTI (ResNet-50)	1024x320	0.082	3.914
KITTI (Efficient-b5)	1024x320	0.080	3.777
CityScapes (ResNet-50)	512x192	0.106	6.237
KITTI (ConvNeXt-L)	1024x320	0.043	1.698

To evaluate a model on KITTI, run:

python evaluate_depth_config.py args_files/hisfog/kitti/resnet_320x1024.txt

Make sure you have first run export_gt_depth.py to extract ground truth files.

To evaluate the ConvNeXt-L model (fine-tuned using metric depth), run:

python3 ./finetune/evaluate_metric_depth.py ./finetune/txt_args/eval/eval_kitti.txt ./conf/cvnXt.txt

And to evaluate a model on Cityscapes, run:

python ./tools/evaluate_depth_cityscapes_config.py args_files/args_res50_cityscapes_finetune_192x640_eval.txt

The ground truth depth files for Cityscapes can be found at HERE, Download this and unzip into splits/cityscapes.

To get ground truth depth for KITTI, run:

python export_gt_depth.py --data_path kitti_data --split eigen
python export_gt_depth.py --data_path kitti_data --split eigen_benchmark

🖼 Inference with your own images

python test_simple_SQL_config.py ./args_files/args_test_simple_kitti_320x1024.txt

hisfog / SfMNeXt-Impl

readme