search

zhangganlin / GlORIE-SLAM

GlORIE-SLAM: Globally Optimized RGB-only Implicit Encoding Point Cloud SLAM
https://ganlinzhang.xyz/GlORIE-SLAM/
Apache License 2.0
89 stars 3 forks source link
bundle-adjustment neural-slam slam

readme

logoGlORIE-SLAM: Globally Optimized RGB-only Implicit Encoding Point Cloud SLAM

Ganlin Zhang1* · Erik Sandström1* · Youmin Zhang2,3 · Manthan Patel1
Luc Van Gool1,4,5 · Martin R. Oswald1,6

1 ETH Zurich, 2 University of Bologna, 3 Rock Universe, 4 KU Leuven, 5 INSAIT, 6 University of Amsterdam
(* Equal Contribution)

Paper | Project Website

teaser_video

GlORIE-SLAM produces accurate dense geometry and camera tracking on large-scale indoor scenes thanks to our deformable scene representation and DSPO layer.

teaser_image

GlORIE-SLAM uses a deformable point cloud as the scene representation and achieves lower trajectory error and higher rendering accuracy compared to competitive approaches.

framework

GlORIE-SLAM Architecture.

Table of Contents
  1. Installation
  2. Run
  3. Acknowledgement
  4. Citation
  5. Contact

Installation

  1. Clone the repo using the --recursive flag 

    git clone --recursive https://github.com/zhangganlin/GlORIE-SLAM.git
cd GlOIRE-SLAM

  2. Creating a new conda environment using the provided glorie_env.yaml file.

    Notice that if you are running GlORIE-SLAM on a cluster GPU without a display, we recommend installing the headless version of Open3D which is required to evaluate the depth L1 metric of the reconstructed mesh. This requires compiling Open3D from scratch. If that, remove the Open3D dependency from the glorie_env.yaml file. The code is tested with Open3D 0.16.0. 

    conda env create -f glorie_env.yaml
conda activate glorie-slam

  3. Compile and install thirdparty libraries.

    pip install thirdparty/evaluate_3d_reconstruction_lib
python setup.py install

  4. Download pretrained model. Download the pretained models from Google Drive, unzip them inside the pretrained folder.

    [Directory structure of pretrianed (click to expand)] 
  .
  └── pretrained
        ├── .gitkeep
        ├── droid.pth
        ├── middle_fine.pt
        └── omnidata_dpt_depth_v2.ckpt

Data Download

Replica

Download the data as below and the data is saved into the ./datasets/Replica folder. Note that the Replica data is generated by the authors of iMAP (but hosted by the authors of NICE-SLAM). Please cite iMAP if you use the data.

bash scripts/download_replica.sh

To be able to evaluate the reconstruction error, download the ground truth Replica meshes where unseen region have been culled.

bash scripts/download_cull_replica_mesh.sh

TUM-RGBD

bash scripts/download_tum.sh

DATAROOT is ./datasets by default. Please change the input_folder path in the scene specific config files if stored somewhere else on your machine.

ScanNet

Please follow the data downloading procedure on the ScanNet website, and extract color/depth frames from the .sens file using this code.

[Directory structure of ScanNet (click to expand)] DATAROOT is `./datasets` by default. If a sequence (`sceneXXXX_XX`) is stored in other places, please change the `input_folder` path in the config file or in the command line. ``` DATAROOT └── scannet └── scene0000_00 └── frames ├── color │ ├── 0.jpg │ ├── 1.jpg │ ├── ... │ └── ... ├── depth │ ├── 0.png │ ├── 1.png │ ├── ... │ └── ... ├── intrinsic └── pose ├── 0.txt ├── 1.txt ├── ... └── ... ```

We use the following sequences: 

scene0000_00
scene0054_00
scene0059_00
scene0106_00
scene0169_00
scene0181_00
scene0207_00
scene0233_00

Run

For running GlORIE-SLAM, we recommend using weights and biases for the logging. This can be turned on by setting the wandb flag to True in the configs/*.yaml file. Also make sure to specify the path wandb_folder. If you don't have a wandb account, first create one. Each scene has a config folder, where the input_folder,output paths need to be specified. Below, we show some example run commands for one tiny scene from each dataset.

Replica

To run GlORIE-SLAM on the office0 scene, run the following command. 

python run.py configs/Replica/demo_office0.yaml

After reconstruction, the trajectory error will be evaluated and so will the mesh accuracy along with the rendering metrics.

TUM-RGBD

To run GlORIE-SLAM on the freiburg3_office scene, run the following command. 

python run.py configs/TUM_RGBD/demo_freiburg3_office.yaml

After reconstruction, the trajectory error will be evaluated automatically.

ScanNet

To run GlORIE-SLAM on the scene0000_00 scene, run the following command. 

python run.py configs/Scannet/demo_scene0000.yaml

After reconstruction, the trajectory error will be evaluated automatically.

Run tracking without mapping

Our GlORIE-SLAM pipeline uses two processes for tracking and mapping, and it is possible to run tracking only without mapping/rendering. In case you only want to test the DSPO part of our paper, add --only_trackingin each of the above commands.

python run.py configs/Replica/demo_office0.yaml --only_tracking
python run.py configs/TUM_RGBD/demo_freiburg3_office.yaml --only_tracking
python run.py configs/Scannet/demo_scene0000.yaml --only_tracking

Acknowledgement

Our codebase is partially based on Point-SLAM, GO-SLAM and DROID-SLAM, we thank the authors for making these codebases publicly available. Our work would not have been possible without your great efforts!

Reproducibility

There may be minor differences between the released codebase and the results reported in the paper. Further, we note that the GPU hardware has an influence, despite running the same seed and conda environment.

Citation

If you find our code or paper useful, please cite

@article{zhang2024glorie,
  title={Glorie-slam: Globally optimized rgb-only implicit encoding point cloud slam},
  author={Zhang, Ganlin and Sandstr{\"o}m, Erik and Zhang, Youmin and Patel, Manthan and Van Gool, Luc and Oswald, Martin R},
  journal={arXiv preprint arXiv:2403.19549},
  year={2024}
}

Contact

Contact Ganlin Zhang and Erik Sandström for questions, comments and reporting bugs.