MCC-HO: Multiview Compressive Coding for Hand-Object 3D Reconstruction

Code repository for the paper: Reconstructing Hand-Held Objects in 3D

Jane Wu, Georgios Pavlakos, Georgia Gkioxari, Jitendra Malik

Installation

Installation and preparation follow the MAE repo, just like MCC. Please also install PyTorch3D for 3D related funcionality and the following libraries:

pip install omegaconf trimesh

Data

Please see DATASET.md for information on data preparation.

Demo

To run MCC-HO inference on any input image and estimated 3D hand, please use, e.g.,

python demo.py \
    --image demo/boardgame_v_W_qdSiPKSdQ_frame000019.jpg \
    --hand demo/boardgame_v_W_qdSiPKSdQ_frame000019_hand.obj \
    --seg demo/boardgame_v_W_qdSiPKSdQ_frame000019_mask.png \
    --cam demo/camera_intrinsics_mow.json \
    --checkpoint [path to model checkpoint]

One may use a checkpoint from the training step below or download our pretrained model (trained on DexYCB, MOW, and HOI4D) [here]. One may set the --score_thresholds argument to specify the score thresholds (More points are shown with a lower threshold, but the predictions might be noisier). The script will generate an html file showing an interactive visualizaion of the MCC-HO output with plotly.

Training

To train an MCC-HO model, please run

OUTPUT=model_outputs
python main_mccho.py \
    --mccho_path [path to MCC-HO preprocessed data] \
    --dataset_cache [path to dataset cache] \
    --job_dir $OUTPUT \
    --output_dir $OUTPUT/log \
    --shuffle_train

• Optional: MCC-HO (excluding the segmentation output layers) may be initialized using MCC pre-trained. A pretrained MCC model is available [here].

Acknowledgements

This implementation builds on the MCC codebase, which in turn is based on MAE.

Citing

If you find this code useful for your research, please consider citing the following paper:

@article{wu2024reconstructing,
  title={Reconstructing Hand-Held Objects in 3D},
  author={Wu, Jane and Pavlakos, Georgios and Gkioxari, Georgia and Malik, Jitendra},
  journal={arXiv preprint arXiv:2404.06507,
  year={2024},
}