We propose SurMo, a new paradigm for learning dynamic human rendering from videos by jointly modeling the temporal motion dynamics and human appearances in a unified framework based on a novel surface-based triplane. We extend the existing well-adopted paradigm of "Pose Encoding → Appearance Decoding" to "Motion Encoding → Physical Motion Decoding, Appearance Decoding".
This repository contains the code of SurMo that is built upon HVTR and HVTR++.
SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering
Tao Hu, Fangzhou Hong, Ziwei Liu
CVPR 2024
[Project Page] [Video] [Paper]
HVTR++: Image and Pose Driven Human Avatars using Hybrid Volumetric-Textural Rendering
Tao Hu, Hongyi Xu, Linjie Luo, Tao Yu, Zerong Zheng, He Zhang, Yebin Liu, Matthias Zwicker
TVCG 2023
[Project Page] [Video] [Paper]
HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars
Tao Hu, Tao Yu, Zerong Zheng, He Zhang, Yebin Liu, Matthias Zwicker
3DV 2022
[Project Page] [Video] [Paper]
To facilitate comparisons with our model in subsequent work, we have saved our rendering results on ZJU-MoCap. [SurMo] [HVTR++] [HVTR]
NVIDIA GPUs are required for this project. We have trained and tested code on NVIDIA V100. We recommend using anaconda to manage the python environments.
conda create --name surmo python=3.9
conda install pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.1 -c pytorch
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d
pip install -r requirements.txt
Download the pre-trained models and assets. Put them in DATA_DIR/result/trained_model and DATA_DIR/asset respectively, or download models via scripts python download_models.py. DATA_DIR is specified as ../data in default.
Download ZJU-MoCap dataset and put it in the folder zju_mocap (e.g., DATA_DIR/zju_mocap/CoreView_3XX)
Register and download SMPL models here. Put them in the folder smpl_data.
The folder structure should look like
DATA_DIR
├── zju_mocap
└── asset/
├── smpl_data/
└── SMPL_NEUTRAL.pkl
├── result/trained_model
Inference script for models (313, 315, 377, 386, 387, 394) trained on ZJU-MoCap.
bash scripts/3XX_inference.sh gpu_ids
e.g., bash scripts/313_inference.sh 0
The inference results will be found in DATA_DIR/result/.
Training script for subjects (313, 315, 377, 386, 387, 394) on ZJU-MoCap.
bash scripts/3XX_train.sh gpu_ids
e.g., bash scripts/313_train.sh 0,1,2,3
The trained models will be found in DATA_DIR/result/trained_model/.
Distributed under the S-Lab License. See LICENSE
for more information.
@misc{hu2024surmo,
title={SurMo: Surface-based 4D Motion Modeling for Dynamic Human Rendering},
author={Tao Hu and Fangzhou Hong and Ziwei Liu},
year={2024},
eprint={2404.01225},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
@ARTICLE{hu2023hvtrpp,
author={Hu, Tao and Xu, Hongyi and Luo, Linjie and Yu, Tao and Zheng, Zerong and Zhang, He and Liu, Yebin and Zwicker, Matthias},
journal={IEEE Transactions on Visualization and Computer Graphics},
title={HVTR++: Image and Pose Driven Human Avatars using Hybrid Volumetric-Textural Rendering},
year={2023}
}
@inproceedings{hu2022hvtr,
title={HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars},
author={Hu, Tao and Yu, Tao and Zheng, Zerong and Zhang, He and Liu, Yebin and Zwicker, Matthias},
booktitle = {2022 International Conference on 3D Vision (3DV)},
year = {2022}
}