This is the official implementation of Neuralangelo: High-Fidelity Neural Surface Reconstruction.
Zhaoshuo Li,
Thomas Müller,
Alex Evans,
Russell H. Taylor,
Mathias Unberath,
Ming-Yu Liu,
Chen-Hsuan Lin
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023
The code is built upon the Imaginaire library from the Deep Imagination Research Group at NVIDIA.
For business inquiries, please submit the NVIDIA research licensing form.
We offer two ways to setup the environment:
We provide prebuilt Docker images, where
docker.io/chenhsuanlin/colmap:3.8
is for running COLMAP and the data preprocessing scripts. This includes the prebuilt COLMAP library (CUDA-supported).docker.io/chenhsuanlin/neuralangelo:23.04-py3
is for running the main Neuralangelo pipeline.The corresponding Dockerfiles can be found in the docker
directory.
neuralangelo
with
conda env create --file neuralangelo.yaml
conda activate neuralangelo
For COLMAP, alternative installation options are also available on the COLMAP website.
Please refer to Data Preparation for step-by-step instructions.
We assume known camera poses for each extracted frame from the video.
The code uses the same json format as Instant NGP.
EXPERIMENT=toy_example
GROUP=example_group
NAME=example_name
CONFIG=projects/neuralangelo/configs/custom/${EXPERIMENT}.yaml
GPUS=1 # use >1 for multi-GPU training!
torchrun --nproc_per_node=${GPUS} train.py \
--logdir=logs/${GROUP}/${NAME} \
--config=${CONFIG} \
--show_pbar
Some useful notes:
--wandb
to the command line argument to enable W&B logging.--wandb_name
to specify the W&B project name.init_wandb()
function in imaginaire/trainers/base.py
.--optim.params.lr=1e-2
).--checkpoint={CHECKPOINT_PATH}
to initialize with a certain checkpoint; set --resume
to resume training.data.num_images
is set to the number of training images.Use the following command to run isosurface mesh extraction:
CHECKPOINT=logs/${GROUP}/${NAME}/xxx.pt
OUTPUT_MESH=xxx.ply
CONFIG=logs/${GROUP}/${NAME}/config.yaml
RESOLUTION=2048
BLOCK_RES=128
GPUS=1 # use >1 for multi-GPU mesh extraction
torchrun --nproc_per_node=${GPUS} projects/neuralangelo/scripts/extract_mesh.py \
--config=${CONFIG} \
--checkpoint=${CHECKPOINT} \
--output_file=${OUTPUT_MESH} \
--resolution=${RESOLUTION} \
--block_res=${BLOCK_RES}
Some useful notes:
--textured
to extract meshes with textures.--keep_lcc
to remove noises. May also remove thin structures.BLOCK_RES
to reduce GPU memory usage.RESOLUTION
to reduce mesh size.Q: CUDA out of memory. How do I decrease the memory footprint?
A: Neuralangelo requires at least 24GB GPU memory with our default configuration. If you run out of memory, consider adjusting the following hyperparameters under model.object.sdf.encoding.hashgrid
(with suggested values):
GPU VRAM | Hyperparameter |
---|---|
8GB | dict_size=20 , dim=4 |
12GB | dict_size=21 , dim=4 |
16GB | dict_size=21 , dim=8 |
Please note that the above hyperparameter adjustment may sacrifice the reconstruction quality.
If Neuralangelo runs fine during training but CUDA out of memory during evaluation, consider adjusting the evaluation parameters under data.val
, including setting smaller image_size
(e.g., maximum resolution 200x200), and setting batch_size=1
, subset=1
.
Q: The reconstruction of my custom dataset is bad. What can I do?
A: It is worth looking into the following:
If you find our code useful for your research, please cite
@inproceedings{li2023neuralangelo,
title={Neuralangelo: High-Fidelity Neural Surface Reconstruction},
author={Li, Zhaoshuo and M\"uller, Thomas and Evans, Alex and Taylor, Russell H and Unberath, Mathias and Liu, Ming-Yu and Lin, Chen-Hsuan},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2023}
}