Chenxing Jiang*, Hanwen Zhang*, Peize Liu, Zehuan Yu, Hui Cheng, Boyu Zhou †, Shaojie Shen
IEEE Robotics and Automation Letters (2023 Best Paper Award)
Constructing a high-quality dense map in real-time is essential for robotics, AR/VR, and digital twins applications. As Neural Radiance Field (NeRF) greatly improves the mapping performance, in this letter, we propose a NeRF-based mapping method that enables higher-quality reconstruction and real-time capability even on edge computers. Specifically, we propose a novel hierarchical hybrid representation that leverages implicit multiresolution hash encoding aided by explicit octree SDF priors, describing the scene at different levels of detail. This representation allows for fast scene geometry initialization and makes scene geometry easier to learn. Besides, we present a coverage-maximizing keyframe selection strategy to address the forgetting issue and enhance mapping quality, particularly in marginal areas. To the best of our knowledge, our method is the first to achieve high-quality NeRF-based mapping on edge computers of handheld devices and quadrotors in real-time. Experiments demonstrate that our method outperforms existing NeRF-based mapping methods in geometry accuracy, texture realism, and time consumption.
Begin by cloning this repository and all its submodules using the following command:
git clone --recursive https://github.com/SYSU-STAR/H2-Mapping
Install the Pytorch manually for your hardware platform.
Create an anaconda environment called h2mapping
. Note that installing the 0.17.0 version of open3d may result in errors during reconstruction evaluation. Please install the 0.16.0 version of open3d instead.
cd H2-Mapping/mapping
conda env create -f h2mapping.yaml
bash install.sh
cd third_party/tinycudann
cmake . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build --config RelWithDebInfo -j
cd bindings/torch
python setup.py install
src/mapping.py
with the built librarytorch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
./Datasets/Replica
folder.bash mapping/scripts/download_replica.sh
configs/replica/room_0
.# take replica room0 dataset as example
cd mapping
python -W ignore demo/run_mapping.py configs/replica/room_0.yaml
The final reconstructed mesh will be saved in mapping/logs/{DATASET}/{DATA SEQUENCE}/{FILE_NAME}/mesh
.
Install the Ubuntu, ROS, Ceres, OpenCV. If you successfully run VINS-Fusion, you will be able to run our tracking code as well.
The current version of our platform includes:
Ubuntu=20.04
ROS=noetic
Ceres=1.14.0
OpenCV=4.2.0
Build our Tracking module
cd H2-Mapping
catkin_make
src/mapping.py
with the built librarytorch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
Please download the ROS bag containing RGB-D sensor data from the Realsense L515. You can access the file through the provided link: tower_compress.bag;
Decompress the ROS bag
rosbag decompress tower_compress.bag
Configure the tracking parameter in the designated configuration file. For reference, consider the example file located at src/dvins/config/uav2022/uav_nerf.yaml
. This configuration file is for the provided tower_compress.orig.bag
.
Configure the ros_args
parameter in the designated configuration file. For reference, consider the example file located at configs/realsense/realsense.yaml
. This configuration file is for the provided tower_compress.orig.bag
.
# set intrinsic parameters, ROS topics of rgb image, depth image and odometry
ros_args:
intrinsic: [601.347290039062, 601.343017578125, 329.519226074219, 238.586654663086] # K[0, 0], K[1, 1], K[0, 2], K[1, 2]
color_topic: '/camera/color/image_raw'
depth_topic: '/camera/aligned_depth_to_color/image_raw'
pose_topic: '/vins_estimator/cam_pose'
Run the mapping module. If you wish to preserve the intermediate results for visualization purposes, you have the option to modify the "save_ckpt_freq" parameter in the configuration file configs/realsense/tower
.
conda activate h2mapping
cd H2-Mapping
source devel/setup.bash
cd mapping
python -W ignore demo/run_mapping.py configs/realsense/tower.yaml -run_ros
Once the mapping module is prepared and operational, you will observe a distinctive sign in the console that reads " ========== MAPPING START ==========="
. This indicator confirms the initiation of the mapping process.
Note: The default configuration was utilized in our paper's experiment, conducted on NVIDIA Jetson AGX Orin (32GB RAM). In case you encounter memory limitations on your platform, you can attempt reducing the insert_ratio
parameter in the configs/realsense/tower.yaml
file, but it may result in inferior outcomes.
Run the Tracking module
cd H2-Mapping
bash ros_cmd/run_vins_rgbd.sh
rosbag play tower_compress.orig.bag
rosnode kill -a
to terminate all the modules. Afterwards, the marching cube algorithm will be executed to reconstruct a mesh for visualization.Modify the mapping configuration file in mapping/configs/realsense
. Please take note of the following:
Note:
(1) Ensure that the value assigned to the "offset" (m) parameter is sufficiently large. This value is utilized to ensure that the coordinates of each point are positive.
(2) Ensure that the upper bound of the "bound" (m) parameter is suitably large. This parameter defines the boundary of the scene. However, if the range is too much larger than the scene you want to reconstruct, the performance may degrade.
(3) Choose the appropriate value for "num_vertexes" based on the size of your scene. It should be large enough to encompass all the vertices of the octree.
(4) Ensure that there will be no frame with a minimum depth smaller than max_depth.
Modify the tracking configuration file in src/dvins/config/uav2022/uav_nerf.yaml
to suit your specific device.
To execute the code similar to the provided demo.
bash scripts/download_replica_mesh.sh
eval/eval_recon.py
with the built librarytorch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
-2d
to enable 2D metric. Use -3d
to enable 3D metric. The reconstruction results will be saved in the $OUTPUT_FOLDER
# assign any output_folder and gt mesh you like, here is just an example
cd mapping
OUTPUT_FOLDER=logs/replica/room0/result
GT_MESH=../Datasets/Replica/cull_replica_mesh/room0.ply
python eval/eval_recon.py \
$OUTPUT_FOLDER/bak/config.yaml \
--rec_mesh $OUTPUT_FOLDER/mesh/final_mesh.ply \
--gt_mesh $GT_MESH \
--ckpt $OUTPUT_FOLDER/ckpt/final_ckpt.pth \
--out_dir $OUTPUT_FOLDER \
-2d \
-3d
src/mapping.py
with the built librarytorch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
$OUTPUT_FOLDER
.# assign any output_folder you like, here is just an example
cd mapping
OUTPUT_FOLDER=logs/replica/room0/result
python eval/eval_color.py \
$OUTPUT_FOLDER/bak/config.yaml \
--result_file $OUTPUT_FOLDER
Note: If you want to evaluate the rendering error in your custom dataset, you should emulate the structure of mapping/src/dataset/replica.py
and create a corresponding MY_DATA.py
file. Additionally, ensure to include lines in the config file specifying the dataset as dataset: MY_DATA
and the data_path: YOUR_DATASET_PATH
.
Provided that you have configured the saving of intermediate results by adjusting the "save_ckpt_freq" parameter in the configuration file, you can employ the code to visualize the ongoing process. This functionality allows you to observe the progression of the intermediate results.
vis/vis_mesh.py
with the built librarytorch.classes.load_library("third_party/sparse_octree/build/lib.xxx/svo.xxx.so")
# assign any output_folder you like, here is just an example
# create_mesh is only require to set once
cd mapping
OUTPUT_FOLDER=logs/replica/room0/result
python vis/vis_mesh.py \
$OUTPUT_FOLDER/bak/config.yaml \
--result_file $OUTPUT_FOLDER \
-create_mesh
This issue is related to the version of cv_bridge. To resolve it, you can follow these steps (using ROS Noetic and OpenCV 4.5.2 as an example):
(1) Remove the previous cv_bridge package for the ROS version:
sudo apt-get remove ros-noetic-cv-bridge
(2) Download the new version of cv_bridge
(3) Locate the CMakeLists.txt file of cv_bridge and modify the following line to match the version of OpenCV you have installed
// In CMakeLists.txt of cv_bridge
find_package(OpenCV 4.5.2 REQUIRED) // Change it to your installed version of OpenCV
(4) Build and install the cv_bridge
mkdir build
cd build
cmake ..
make
sudo make install
(5) Add the cmake path for cv_bridge in the src/dvins/vins_estimator/CMakeLists.txt
and src/dvins/pose_graph/CMakeLists.txt
set(cv_bridge_DIR /usr/local/share/cv_bridge/cmake) // At the beginning
We adapted some codes from some remarkable repositories including VINS-Fusion, NICE-SLAM, Vox-Fusion and Tiny-cuda-nn. We express our gratitude for the authors' generosity in sharing their code publicly.
You can contact the author through email: cjiangan@connect.ust.hk, zhanghw66@mail2.sysu.edu.cn and zhouby23@mail.sysu.edu.cn
If you find our work useful, please consider citing:
@ARTICLE{10243098,
author={Jiang, Chenxing and Zhang, Hanwen and Liu, Peize and Yu, Zehuan and Cheng, Hui and Zhou, Boyu and Shen, Shaojie},
journal={IEEE Robotics and Automation Letters},
title={H$_{2}$-Mapping: Real-Time Dense Mapping Using Hierarchical Hybrid Representation},
year={2023},
volume={8},
number={10},
pages={6787-6794},
doi={10.1109/LRA.2023.3313051}}