Xu He1
·
Xiaoyu Li2
·
Di Kang2
·
Jiangnan Ye1
·
Chaopeng Zhang2
·
Liyang Chen1
·
Xiangjun Gao3
·
Han Zhang4
·
Zhiyong Wu1,5
·
Haolin Zhuang1
·
1Tsinghua University 2Tencent AI Lab
3The Hong Kong University of Science and Technology
4Standford University 5The Chinese University of Hong Kong
Model | Resolution | #Views | GPU Memery (w/ refinement) |
#Training Scans | Datasets |
---|---|---|---|---|---|
magicman_base | 512x512 | 20 | 23.5GB | ~2500 | THuman2.1 |
magicman_plus | 512x512 | 24 | 26.5GB | ~5500 | THuman2.1, CustomHumans, 2K2K, CityuHuman |
Currently, we provide two versions of models: a base model trained on ~2500 scans to generate 20 views and an enhanced model trained on ~5500 scans to generate 24 views.
Models can be downloaded here. Both pretrained_weights
and one version of magicman_{version}
are needed to be downloaded and put under ./ckpt
as:
|--- ckpt/
| |--- pretrained_weights/
| |--- magic_base/ or magic_plus/
git clone https://github.com/thuhcsi/MagicMan.git
cd MagicMan
# Create conda environment
conda create -n magicman python=3.10
conda activate magicman
# Install PyTorch and other dependencies
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2
pip install -r requirements.txt
# Install PyTorch3D
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
# Install mmcv-full
pip install "mmcv-full>=1.3.17,<1.6.0" -f https://download.openmmlab.com/mmcv/dist/cu117/torch2.0.1/index.html
# Install mmhuman3d
pip install "git+https://github.com/open-mmlab/mmhuman3d.git"
Register at ICON's website.
Click Register now on all dependencies, then you can download them all using ONE account with:
cd ./thirdparties/econ
bash fetch_data.sh
Requied models and extra data are from SMPL-X, PIXIE, PyMAF-X, and ECON.
python inference.py --config configs/inference/inference-{version}.yaml --input_path {input_image_path} --output_path {output_dir_path} --seed 42 --device cuda:0
# e.g.,
python inference.py --config configs/inference/inference-base.yaml --input_path examples/001.jpg --output_path examples/001 --seed 42 --device cuda:0
Our code follows several excellent repositories. We appreciate them for making their codes available to the public.
If you find our work useful, please consider citing:
@misc{he2024magicman,
title={MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement},
author={Xu He and Xiaoyu Li and Di Kang and Jiangnan Ye and Chaopeng Zhang and Liyang Chen and Xiangjun Gao and Han Zhang and Zhiyong Wu and Haolin Zhuang},
year={2024},
eprint={2408.14211},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
⚠️This is an open-source research exploration rather than a commercial product, so it may not meet all your expectations. Due to the variability of the diffusion model, you may encounter failure cases. Try using different seeds and adjusting the denoising steps if the results are not desirable. Users are free to create novel views using this tool, but they must comply with local laws and use it responsibly. The developers do not assume any responsibility for potential misuse by users.