AutoRecon: Automated 3D Object Discovery and Reconstruction

Project Page | Paper

teaser

AutoRecon: Automated 3D Object Discovery and Reconstruction
Yuang Wang, Xingyi He, Sida Peng, Haotong Lin, Hujun Bao, Xiaowei Zhou
CVPR 2023

About

This is a refactor version of the AutoRecon project based on the NeRFStudio and the SDFStudio codebase. We separate the project into two parts. The coarse decomposition part is implemented in the AutoDecomp repo, which can be used as a general tool for 3D object discovery and preprocessing casual captures for object reconstruction. The neural surface reconstruction part is implemented here.

Installation

Please refer to the installation guide for detailed instructions.

Run the pipeline

Run the pipeline with your own data

Here we take a demo data as an example. In this example, we assume only a sequential stream of images is available. You can easily adapt your data to use AutoRecon.

Download the demo data from Google Drive and put it under data (-> data/custom_data_example/...)
Run the pipeline with exps/code-release/run_pipeline_demo_low-res.sh

NOTE: In the demo script, we assume the images come from a sequential video stream and use sequential matching for SfM. If you have unordered images, the default setting might lead to inferior result and it is recommended to use exhaustive matching or vocab-tree instead.

Run the pipeline with annotated data in the IDR format

Download the BlendedMVS data from GoogleDrive and put it under data (-> data/BlendedMVS/...)
Run one of the script in exps/code-release/bmvs

Run the pipeline with CO3D data

Downloadt the CO3D data from GoogleDrive and put it under data (-> data/CO3D_DEMO/...)
Run one of the script in exps/code-release/co3d_demo

Extract Mesh

You can take the following script as a reference to extract mesh:

# Extract mesh with MC
LOG_DIR="path_to_log_dir"

MC_RES=512
MESH_FN="extracted_mesh_res-${MC_RES}.ply"
MESH_PATH="${LOG_DIR}/${MESH_FN}"

ns-extract-mesh \
    --load-config $LOG_DIR/config.yml \
    --load-dir $LOG_DIR/sdfstudio_models \
    --output-path $MESH_PATH \
    --chunk_size 25000 --store_float16 True \
    --resolution $MC_RES \
    --use_train_scene_box True \
    --seg_aware_sdf False \
    --remove_internal_geometry None \
    --remove_non_maximum_connected_components True \
    --close_holes False --simplify_mesh_final False

NOTE: We postprocess the extracted mesh before evaluation by removing possible internal geometries with ambient occlusion. The postprocessing code depends on pymeshlab which only works on some machines. You can remove possible internal geometries manually with ambient occlusion using MeshLab.

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{wang2023autorecon,
  title={AutoRecon: Automated 3D Object Discovery and Reconstruction},
  author={Wang, Yuang and He, Xingyi and Peng, Sida and Lin, Haotong and Bao, Hujun and Zhou, Xiaowei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21382--21391},
  year={2023}
}

Acknowledgement

This code is built upon the awesome projects including nerfstudio, sdfstudio, nerfacc, tyro and more. Thanks for these great projects!

zju3dv / AutoRecon

readme