Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras
Huajian Huang1, Longwei Li2, Hui Cheng2, and Sai-Kit Yeung1
The Hong Kong University of Science and Technology1, Sun Yat-Sen University2
In Proceedings of Computer Vision and Pattern Recognition Conference (CVPR), 2024
#2@XuYinzhe provides a Docker example.
sudo apt install libeigen3-dev libboost-all-dev libjsoncpp-dev libopengl-dev mesa-utils libglfw3-dev libglm-dev
Dependencies | Tested with | ||
---|---|---|---|
OS | Ubuntu 20.04 LTS | Ubuntu 22.04 LTS | Jetpack 5.1.2 |
gcc | 10.5.0 | 11.4.0 | 9.4.0 |
cmake | 3.27.5 | 3.22.1 | 3.26.4 |
CUDA | 11.8 | 11.8 | 11.4 |
cuDNN | 8.9.3 | 8.7.0 | 8.6.0 |
OpenCV built with opencv_contrib and CUDA | 4.7.0 | 4.8.0 | 4.7.0 |
LibTorch <2.1.2 | cxx11-abi-shared-with-deps-2.0.1+cu118 | 2.0.0+nv23.05-cp38-linux_aarch64 | |
(optional) Intel® RealSense™ SDK 2.0 | |||
(Remark) Jetson AGX Orin Developer Kit we used is with 64GB and its power model was set to MAXN. |
The supported LibTorch version is up to 2.1.2.
If you do not have the LibTorch installed in the system search paths for CMake, you need to add additional options to build.sh
help CMake find LibTorch. See build.sh
for details. Otherwise, you can also add one line before find_package(Torch REQUIRED)
of CMakeLists.txt
:
[Option 1] You can download the libtorch, e.g., cu118 and then extract them to the folder ./the_path_to_where_you_extracted_LibTorch
.
# In a Terminal
wget https://download.pytorch.org/libtorch/cu118/libtorch-cxx11-abi-shared-with-deps-2.0.1%2Bcu118.zip -O libtorch-cu118.zip
unzip libtorch-cu118.zip -d ./the_path_to_where_you_extracted_LibTorch
rm libtorch-cu118.zip
# In CMakeLists.txt
set(Torch_DIR ./the_path_to_where_you_extracted_LibTorch/libtorch/share/cmake/Torch)
[Option 2] Conda. If you are using Conda to manage your python packages and have installed compatible Pytorch, you could set the
# [For Jatson Orin] To install Pytorch in Jatson developer kit, you can run the below commands
# export TORCH_INSTALL=https://developer.download.nvidia.com/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl
# pip install --no-cache $TORCH_INSTALL
set(Torch_DIR /the_path_to_conda/python3.x/site-packages/torch/share/cmake/Torch)
Take version 4.7.0 for example, look into OpenCV realeases and opencv_contrib, you will find OpenCV 4.7.0 and the corresponding opencv_contrib 4.7.0, download them to the same directory (for example, ~/opencv
) and extract them. Then open a terminal and run:
cd ~/opencv
cd opencv-4.7.0/
mkdir build
cd build
# The build options we used in our tests:
cmake -DCMAKE_BUILD_TYPE=RELEASE -DWITH_CUDA=ON -DWITH_CUDNN=ON -DOPENCV_DNN_CUDA=ON -DWITH_NVCUVID=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DOPENCV_EXTRA_MODULES_PATH="../../opencv_contrib-4.7.0/modules" -DBUILD_TIFF=ON -DBUILD_ZLIB=ON -DBUILD_JASPER=ON -DBUILD_CCALIB=ON -DBUILD_JPEG=ON -DWITH_FFMPEG=ON ..
# Take a moment to check the cmake output, see if there are any packages needed by OpenCV but not installed on your device
make -j8
# NOTE: We found that the compilation of OpenCV may stuck at 99%, this may be caused by the final linking process. We just waited for a while until it was completed and exited without errors.
To install OpenCV into the system path:
sudo make install
If you prefer installing OpenCV to a custom path by adding -DCMAKE_INSTALL_PREFIX=/your_preferred_path
option to the cmake
command, remember to help Photo-SLAM find OpenCV by adding additional cmake options. See build.sh
for details. Otherwise, you can also add the following line to CMakeLists.txt
, ORB-SLAM3/CMakeLists.txt
and ORB-SLAM3/Thirdparty/DBoW2/CMakeLists.txt
, just like what we did for LibTorch.
set(OpenCV_DIR /your_preferred_path/lib/cmake/opencv4)
git clone https://github.com/HuajianUP/Photo-SLAM.git
cd Photo-SLAM/
chmod +x ./build.sh
./build.sh
The benchmark datasets mentioned in our paper: Replica (NICE-SLAM Version), TUM RGB-D, EuRoC.
(optional) Download the dataset.
cd scripts
chmod +x ./*.sh
./download_replica.sh
./download_tum.sh
./download_euroc.sh
For testing, you could use the below commands to run the system after specifying the PATH_TO_Replica
and PATH_TO_SAVE_RESULTS
. We would disable the viewer by adding no_viewer
during the evaluation.
./bin/replica_rgbd \
./ORB-SLAM3/Vocabulary/ORBvoc.txt \
./cfg/ORB_SLAM3/RGB-D/Replica/office0.yaml \
./cfg/gaussian_mapper/RGB-D/Replica/replica_rgbd.yaml \
PATH_TO_Replica/office0 \
PATH_TO_SAVE_RESULTS
# no_viewer
We also provide scripts to conduct experiments on all benchmark datasets mentioned in our paper. We ran each sequence five times to lower the effect of the nondeterministic nature of the system. You need to change the dataset root lines in scripts/*.sh then run:
cd scripts
chmod +x ./*.sh
./replica_mono.sh
./replica_rgbd.sh
./tum_mono.sh
./tum_rgbd.sh
./euroc_stereo.sh
# etc.
git clone https://github.com/HuajianUP/Photo-SLAM-eval.git
We provide an example with the Intel RealSense D455 at examples/realsense_rgbd.cpp
. Please see scripts/realsense_d455.sh
for running it.
<pre class="citation-code"><code><span>@inproceedings</span>{hhuang2024photoslam,
title = {Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Cameras},
author = {Huang, Huajian and Li, Longwei and Cheng Hui and Yeung, Sai-Kit},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2024}
}