Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichntenhofer, Georgia Gkioxari
This is a PyTorch implementation of Multiview Compressive Coding (MCC):
@article{wu2023multiview,
author = {Wu, Chao-Yuan and Johnson, Justin and Malik, Jitendra and Feichtenhofer, Christoph and Gkioxari, Georgia},
title = {Multiview Compressive Coding for 3{D} Reconstruction},
journal = {arXiv preprint arXiv:2301.08247},
year = {2023},
}
This repo is a modified from the MAE repo. Installation and preparation follow that repo. Please also install PyTorch3D for 3D related funcionality.
pip install h5py omegaconf submitit
Please see DATASET.md for information on data preparation.
To train an MCC model on the CO3D v2 dataset, please run
python submitit_mcc.py \
--use_volta32 \
--job_dir ./output \
--nodes 4 \
--co3d_path [path to CO3D dataset] \
--resume [path to pretrained weights for RGB encoder (optional)] \
--holdout_categories \
--holdout_categories
, we hold out a subset of categories during training, and evaluate on the held out categories only.
To train on all categories, please remove the argument.To train an MCC model on the Hypersim dataset, please run
python submitit_mcc.py \
--use_volta32 \
--job_dir ./output \
--nodes 4 \
--hypersim_path [path to Hypersim dataset] \
--resume [path to pretrained weights for RGB encoder (optional)] \
--use_hypersim \
--viz_granularity 0.2 --eval_granularity 0.2 \
--blr 5e-5 \
--epochs 50 \
--train_epoch_len_multiplier 3200 \
--eval_epoch_len_multiplier 200 \
--use_hypersim
for running Hypersim scene reconstruction experiments.To test on iPhone captures, please use the Record3D App on an iPhone to capture an RGB image and the corresonding point cloud (.obj) file. To generate the segmentation mask, we used a private segmentation model; Users may use other tools/models to obtain the mask. Two example captures are available in the demo
folder.
To run MCC inference on the example, please use, e.g.,
python demo.py --image demo/quest2.jpg --point_cloud demo/quest2.obj --seg demo/quest2_seg.png \
--checkpoint [path to model checkpoint] \
One may use a checkpoint from the training step above or download a model that is already trained on all CO3D v2 categories [here].
One may set the --score_thresholds
argument to specify the score thresholds (More points are shown with a lower threshold, but the predictions might be noisier).
The script will generate an html file showing an interactive visualizaion of the MCC output with plotly.
Part of this implementation is based on the MAE codebase. We thank Sasha Sax for help on loading Hypersim and Taskonomy data.
Multiview Compressive Coding is released under the CC-BY-NC 4.0.