SGAligner : 3D Scene Alignment with Scene Graphs

ICCV 2023

Sayan Deb Sarkar¹, Ondrej Miksik², Marc Pollefeys^1,2, Daniel Barath¹, Iro Armeni^{1 1}ETH Zurich ²Microsoft Mixed Reality & AI Labs SGAligner aligns 3D scene graphs of environments using multi-modal learning and leverage the output for the downstream task of 3D point cloud registration. [](https://paperswithcode.com/sota/3d-scene-graph-alignment-on-3dssg?p=sgaligner-3d-scene-alignment-with-scene) [](https://paperswithcode.com/sota/point-cloud-registration-on-3rscan?p=sgaligner-3d-scene-alignment-with-scene) 

[Project Webpage] [Paper]

News :newspaper:

• 14. July 2023 : SGAligner accepted to ICCV 2023. :fire:
• 1. May 2023: SGAligner preprint released on arXiv.
• 10. April 2023: Code released.

Code Structure :clapper:

├── sgaligner
│   ├── example-data                  <- examples of data generated post preprocessing
│   ├── data-preprocessing            <- subscan generation + preprocessing
│   ├── configs                       <- configuration files
│   ├── src
│   │   │── aligner                   <- SGAligner modules
│   │   │── datasets                  <- dataloader for 3RScan subscans
│   │   │── engine                    <- trainer classes
│   │   │── GeoTransformer            <- geotransformer submodule for registration
│   │   │── inference                 <- inference files for alignment + downstream applications
│   │   │── trainers                  <- train + validation loop (EVA + SGAligner)
│   │── utils                         <- util functions
│   │── README.md                    
│   │── scripts                       <- bash scripts for data generation + preprocesing + training
│   └── output                        <- folder that stores models and logs
│

Dependencies :memo:

The main dependencies of the project are the following:

python: 3.8.15
cuda: 11.6

You can set up a conda environment as follows :

git clone --recurse-submodules -j8 git@github.com:sayands/sgaligner.git
cd sgaligner
conda env create -f req.yml

Note: Please follow PointCloudTransformer repository for installing pointnet2_ops_lib if you want to use PCT as the backbone.

Please follow the submodule for additional installation requirements and setup of GeoTransformer.

Downloads :droplet:

The pre-trained model and other meta files are available here.

Dataset Generation :hammer:

After installing the dependencies, we preprocess the datasets and provide the benchmarks.

Subscan Pair Generation - 3RScan + 3DSSG

Download 3RScan and 3DSSG. Move all files of 3DSSG to a new files/ directory within Scan3R. The structure should be:

├── 3RScan
│   ├── files       <- all 3RScan and 3DSSG meta files (NOT the scan data)  
│   ├── scenes      <- scans
│   └── out         <- Default output directory for generated subscans (created when running pre-processing)

To generate labels.instances.align.annotated.v2.ply for each 3RScan scan, please refer to the repo from here.

Change the absolute paths in utils/define.py.

First, we create sub-scans from each 3RScan scan using the ground truth scene Graphs from the 3DSSG dataset and then calculate the pairwise overlap ratio for the subscans in a scan. Finally, we preprocess the data for our framework. The relevant code can be found in the data-preprocessing/ directory. You can use the following command to generate the subscans.

bash scripts/generate_data_scan3r_gt.sh

Note To adhere to our evaluation procedure, please do not change the seed value in the files in configs/ directory.

Generating Overlapping and Non-Overlapping Subscan Pairs

To generate overlapping and non-overlapping pairs, use :

python preprocessing/gen_all_pairs_fileset.py

This will create a fileset with the same number of randomly chosen non-overlapping pairs from the generated subscans as overlapping pairs generated before during subscan generation.

Usage on Predicted Scene Graphs : Coming Soon!

Training :bullettrain_side:

To train SGAligner on 3RScan subscans generated from here, you can use :

cd src
python trainers/trainval_sgaligner.py --config ../configs/scan3r/scan3r_ground_truth.yaml

EVA Training

We also provide training scripts for EVA, used as a baseline after being adapted for scene graph alignment. To train EVA similar to SGAligner on the same data, you can use :

cd src
python trainers/trainval_eva.py --config ../configs/scan3r/scan3r_eva.yaml

We provide config files for the corresponding data in config/ directory. Please change the parameters in the configuration files, if you want to tune the hyper-parameters.

Evaluation :vertical_traffic_light:

Graph Alignment + Point Cloud Registration

cd src
python inference/sgaligner/inference_align_reg.py --config ../configs/scan3r/scan3r_ground_truth.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

Finding Overlapping vs Non-Overlapping Pairs

:heavy_exclamation_mark: Run Generating Overlapping and Non-Overlapping Subscan Pairs before.

To run the inference, you need to:

cd src
python inference/sgaligner/inference_find_overlapper.py --config ../configs/scan3r/scan3r_gt_w_wo_overlap.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

3D Point Cloud Mosaicking

First, we generate the subscans per 3RScan scan using :

python data-preprocessing/gen_scan_subscan_mapping.py --split <the split you want to generate the mapping for>

And then, to run the inference, you need to:

cd src
python inference/sgaligner/inference_mosaicking.py --config ../configs/scan3r/scan3r_gt_mosaicking.yaml --snapshot <path to SGAligner trained model> --reg_snapshot <path to GeoTransformer model trained on 3DMatch>

Benchmark :chart_with_upwards_trend:

We provide detailed results and comparisons here.

3D Scene Graph Alignment (Node Matching)

3D Point Cloud Registration

TODO :soon:

• [X] Add 3D Point Cloud Mosaicking
• [X] Add Support For EVA
• [ ] Add usage on Predicted Scene Graphs
• [ ] Add scene graph alignment of local 3D scenes to prior 3D maps
• [ ] Add overlapping scene finder with a traditional retrieval method (FPFH + VLAD + KNN)

BibTeX :pray:

@article{sarkar2023sgaligner,
      title={SGAligner : 3D Scene Alignment with Scene Graphs}, 
      author={Sayan Deb Sarkar and Ondrej Miksik and Marc Pollefeys and Daniel Barath and Iro Armeni},
      journal={Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
      year={2023}
}

Acknowledgments :recycle:

In this project we use (parts of) the official implementations of the following works and thank the respective authors for open sourcing their methods:

• SceneGraphFusion (3RScan Dataloader)
• GeoTransformer (Registration)
• MCLEA (Alignment)