search

shad0wta9 / meshavatar

Code Repository for MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos (ECCV 2024)
Other
91 stars 3 forks source link

readme

MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos (ECCV 2024)

Yushuo Chen1, Zerong Zheng2, Zhe Li1, Chao Xu2, Yebin Liu1

1Tsinghua Univserity 2NNKosmos Technology

Projectpage · Paper · Video

teaser

Installation

  1. Install dependencies 
    
conda create -n meshavatar python=3.9
conda activate meshavatar

conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge pip install ninja imageio PyOpenGL glfw xatlas gdown pip install git+https://github.com/NVlabs/nvdiffrast/ pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch imageio_download_bin freeimage

pip install -r requirements.txt


2. Download [SMPL](https://smpl.is.tue.mpg.de/), [SMPL-X](https://smpl-x.is.tue.mpg.de/), and the [model correspondences](https://smpl-x.is.tue.mpg.de/). Organize them in some directory as follow and specify `smpl_dir` in `data_preprocessing/script.py`.

smpl_models/ smpl/ SMPL_NEUTRAL.pkl smplx/ SMPLX_NEUTRAL.pkl SMPLX_NEUTRAL.npz model_transfer/ smplx_to_smpl.pkl


3. Download or build `PointInterpolant`. Compile`lbs_surf_grad`, the c++ program for computing surface gradient of LBS weights.

git clone https://github.com/mkazhdan/PoissonRecon cd PoissonRecon make pointinterpolant cd ..

cd data_preprocessing/diffused_skinning sh compile_lbs_surf_grad.sh cd ../..


# Data Preparation

## Data Organization

### AvatarReX Dataset [[link]](https://github.com/lizhe00/AnimatableGaussians/blob/master/AVATARREX_DATASET.md)
Directly download the dataset. If you want to reproduce the experiments in the paper, please resize the images to `1024x750` first, due to the limited GPU Memory.

### SyntheticHuman++ Dataset [[link]](https://github.com/zju3dv/animatable_nerf)
Since our code only supports SMPL-X model, we have converted the parameters [here](https://drive.google.com/drive/folders/1DRlVlb618zVNfFm7bEgRbafUndJVnrB9?usp=sharing) (using [the official transfer model](https://github.com/vchoutas/smplx/tree/main/transfer_model)).

Run `data_preprocessing/convert_calib.py` to convert the calibration format.

### Custom Dataset
Organize your dataset as AvatarReX dataset:
/ mask/pha/ 00000000.jpg ... 00000000.jpg ... / ... / calibration_full.json smpl_params.npz ``` ## Data Preprocessing Specify `data_dir`, `smpl_file_path` and `dataset_name` in your config file. Then preprocess the dataset by ``` python data_preprocessing/script.py --config ``` # Training Run using single gpu ``` sh run.sh # Example sh run.sh 0 configs/jody.json ``` or using multiple gpus ([apex](https://github.com/NVIDIA/apex) should be additionally installed) ``` sh run_multi.sh (, optional) # Example sh run_multi.sh 0,1 configs/avatarrex_zzr.json ``` # Testing Run ``` python test.py --config out//exp.json ``` Modifiy the arguments in `test.py` to choose any output you want. # Acknowledgement Part of our code is borrowed from the following repos, great thanks to them! * [nvdiffrec](https://github.com/NVlabs/nvdiffrec), [nvdiffrecmc](https://github.com/NVlabs/nvdiffrecmc) * [Pytorch-UNet](https://github.com/milesial/Pytorch-UNet) * [fite](https://github.com/jsnln/fite) # Citation If you find our code or data is helpful to your research, please consider citing our paper. ```bibtex @misc{chen2024meshavatar, title={MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos}, author={Yushuo Chen and Zerong Zheng and Zhe Li and Chao Xu and Yebin Liu}, year={2024}, eprint={2407.08414}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2407.08414}, } ```