3D Semantic Scene Graph Estimations

This is a framework for developing 3D semantic scene graph estimations. The repository includes five different methods, namely IMP, VGfM, 3DSSG, SGFN and MonoSSG.

This repository has been used for the following publications:

* **Title**: [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences][MonoSSG] **Authors**: [Shun-Cheng Wu][sc], [Keisuke Tateno][keisu], [Nassir Navab][nassir] and [Federico Tombari][fede] * **Title**: [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences][SGFN] **Authors**: [Shun-Cheng Wu][sc], [Johanna Wald][jojo], [Keisuke Tateno][keisu], [Nassir Navab][nassir] and [Federico Tombari][fede] [sc]:http://campar.in.tum.de/Main/ShunChengWu [keisu]:http://campar.in.tum.de/Main/KeisukeTateno [jojo]:http://campar.in.tum.de/Main/JohannaWald [nassir]:http://campar.in.tum.de/Main/NassirNavabCv [fede]:http://campar.in.tum.de/Main/FedericoTombari

Setup

Environment.

# if you don't have miniconda
source setup_conda.sh 

# setup
source setup.sh

mkdir data
ln -s /path/to/your/3RScan ./data/

source Init.sh # This will set PYTHONPATH and activate the environment for you.

Preparation

Download data

cd files
bash preparation.sh

Prepare 3RScan dataset

Please make sure you agree the 3RScan Terms of Use first, and get the download script and put it right at the 3RScan main directory.

Then run

python scripts/RUN_prepare_dataset_3RScan.py --download --thread 8

Generate Experiment data

# For GT
# This script downloads preprocessed data for GT data generation, and generate GT data.
python scripts/RUN_prepare_GT_setup_3RScan.py --thread 16

# For Dense
# This script downloads the inseg.ply files and unzip them to your 3rscan folder, and 
generates training data.
python scripts/RUN_prepare_Dense_setup_3RScan.py -c configs/dataset/config_base_3RScan_inseg_l20.yaml --thread 16

# For Sparse
# This script downloads the 2dssg_orbslam3.[json,ply] files and unzip them to your 3rscan folder, and 
generates training data.
python scripts/RUN_prepare_Sparse_setup_3RScan.py -c configs/dataset/config_base_3RScan_orbslam_l20.yaml --thread 16

Train

The first time you may need want to chagne the wandb account in configs/config_default.yaml. Change the wanb.entity and wanb.project to yours. Or you can disable logging by passing --dry_run.

source Init.sh

# Train and eval everything. 
python scripts/RUN_traineval_all.py

# Train single
python main.py --mode train --config /path/to/your/config/file

# Eval one
python main.py --mode eval --config /path/to/your/config/file

Trained models

We provide trained model using the optimized code (this one), instead of the one reported in our CVPR23 paper. Although the numbers are different but all methods follow the same trend. We encourage people compare to the results obtained by yourself using this repo.

Download the trained models and unzip them under experiments folder (you may need to create one by yourself).

Note that the implementation of SceneGraphFusion and 3DSSG are different from the original papers. We tried to make all methods shared the same model settings in order to compare them fairly.

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Citations

If you find the code useful please consider citing our papers:

@inproceedings{wu2023incremental,
  title={Incremental 3D Semantic Scene Graph Prediction from RGB Sequences},
  author={Wu, Shun-Cheng and Tateno, Keisuke and Navab, Nassir and Tombari, Federico},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5064--5074},
  year={2023},
}

@inproceedings{wu2021scenegraphfusion,
  title={Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences},
  author={Wu, Shun-Cheng and Wald, Johanna and Tateno, Keisuke and Navab, Nassir and Tombari, Federico},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={7515--7525},
  year={2021}
}

@inproceedings{Wald2020,
    title = {{Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions}},
    author = {Wald, Johanna and Dhamo, Helisa and Navab, Nassir and Tombari, Federico},
    booktitle = {Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2020}
}