Welcome to the official repository for PhotoReg, a project dedicated to the registration of 3D Gaussian Splatting models.
Begin by cloning the required repositories. You'll need gaussian-splatting
for preprocessing datasets and preparing input 3DGS models:
git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
Then, clone the PhotoReg repository:
git clone --recursive https://github.com/ziweny11/PhotoRegCodes
Navigate to the PhotoReg project directory and set up the environment:
cd PhotoRegCodes
conda create -n photoreg python=3.8 cmake=3.14.0
conda activate photoreg
conda install pytorch torchvision torchaudio cuda-toolkit=11.8 -c pytorch -c nvidia
conda install -c conda-forge numpy plyfile tqdm matplotlib scipy einops trimesh tensorboard "pyglet<2" open3d pip
pip install roma opencv-python e3nn pytorch3d submodules/diff-gaussian-rasterization submodules/simple-knn
For the diff-gaussian-rasterization-depth
package:
git clone https://github.com/leo-frank/diff-gaussian-rasterization-depth.git
cd diff-gaussian-rasterization-depth
python setup.py install
Download the pretrained model for DUSt3R here and place it in the PhotoRegCodes
folder. Learn more about DUSt3R on their GitHub page.
Download the example dataset from this Google Drive link and place it in the PhotoRegCodes
folder.
Run the demo with:
python main.py -s data/truckright3 --start_checkpoint data/truckright3_model/chkpnt10000.pth --start_checkpoint2 data/truckleft3_model/chkpnt10000.pth --campose1 data/truckright3 --campose2 data/truckleft3
To visualize the final fused GS model (gnew.pth
), first run:
python realtime_gsviz.py -s data/truckright3/ --start_checkpoint data/truckright3_model/chkpnt10000.pth
Then, use the SIBR viewer from gaussian-splatting
:
./<SIBR install dir>/bin/SIBR_remoteGaussian_app
For installation details and more information, refer to the gaussian-splatting README.
Use the following command structure for general datasets:
python main.py -s path/to/source1 --start_checkpoint path/to/model/chkpntXXX.pth --start_checkpoint2 path/to/model2/chkpntXXX.pth --campose1 path/to/source1 --campose2 path/to/source2
path/to/source
are paths to the
<location>
|---images
| |---<image 0>
| |---<image 1>
| |---...
|---sparse
|---0
|---cameras.bin
|---images.bin
|---points3D.bin
path/to/model/chkpntXXX.pth
are paths to the output 3DGS model. By default, it is in the output folder in gaussian-splatting after training, where the chkpntXXX.pth is stoed in folder
structured like:
<location>
|---point_cloud
| |---...
|---cameras.json
|---cfg_args
|---input.ply
|---chkpntXXX.pth
You can visualize the result gnew.pth using SIBR the same as in demo.
If you find our work useful, please consider citing:
@misc{yuan2024photoreg,
title={PhotoReg: Photometrically Registering 3D Gaussian Splatting Models},
author={Ziwen Yuan and Tianyi Zhang and Matthew Johnson-Roberson and Weiming Zhi},
year={2024},
eprint={2410.05044},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2410.05044}
}