AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling

Ziqian Bai · Timur Bagautdinov · Javier Romero . Michael Zollhöfer · Ping Tan · Shunsuke Saito

ECCV 2022

AutoAvatar is an autoregressive approach for modeling dynamically deforming human bodies directly from raw scans without the need of precise surface registration.

Data Preparation of DFaust

Create "DFaust" folder under "\<workspace_folder>".

cd <workspace_folder>
mkdir DFaust

Download SMPL+H parameters of DFaust from AMASS dataset to "\<workspace_folder>/DFaust". Unzip to get the "DFaust_67" folder.
Download Dfaust scan data from link. Here, we take subject 50002 as an example in the following steps. Unzip data to "\<workspace_folder>/DFaust/scans/50002".
Download SMPL model from link. Download SMPL meta data from link. Move SMPL related files "basicmodel_m_lbs_10_207_0_v1.0.0.pkl", "basicModel_f_lbs_10_207_0_v1.0.0.pkl", "uv_info.npz", and "smpl_resample_idxs.npz" into "\<workspace_folder>/SMPL".
Set up AMASS for DFaust data preprocessing. More specifically, download SMPL+H (smplh.tar.xz) and unzip to "\<workspace_folder>/SMPL/smplh". Download DMPLs (dmpls.tar.xz) and unzip to "\<workspace_folder>/SMPL/dmpls".
clone this repo to "\<workspace_folder>".

cd <workspace_folder>
git clone https://github.com/facebookresearch/AutoAvatar.git

Now we should have the following folder structure:

    \<workspace_folder\>
    ├── DFaust
    │   ├── DFaust_67
    │   │   └── 50002
    │   │       └── *.npz
    │   └── scans
    │       └── 50002
    │           └── \<sequences_folders\>
    ├── SMPL
    │   ├── smplh
    │   │   ├── female
    │   │   ├── male
    │   │   └── neutral
    │   ├── dmpls
    │   │   ├── female
    │   │   ├── male
    │   │   └── neutral
    |   └── \<other_SMPL_related_files\>
    └── AutoAvatar

Environment Setup

Install Anaconda or Miniconda. Then run the setup script.

cd AutoAvatar
conda create -n AutoAvatar python=3.8
conda activate AutoAvatar
bash setup.sh

Create "external" folder and install human_body_prior for DFaust data preprocess.

mkdir external
cd external
git clone https://github.com/nghorbani/human_body_prior.git
cd human_body_prior
python setup.py develop

Data Preprocess

Run "DFaust_generate.py" to preprocess data. Note that this may take a long time due to the mesh simplification (the open3d API mesh_o3d.simplify_quadric_decimation() in simplify_scans())! Mesh simplification is to speed up data loading during training.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python data/DFaust_generate.py --ws_dir <workspace_folder>

Train

Run "implicit_train_dfaust.py" to train the model.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python exps/PosedDecKNN_dPoses_dHs/implicit_train_dfaust.py --ws_dir <workspace_folder> --configs_path configs/PosedDecKNN_dPoses_dHs/AutoRegr.yaml --configs_path_rollout configs/PosedDecKNN_dPoses_dHs/AutoRegr_Rollout2.yaml

Test

Run "implicit_eval_dfaust.py" to test the model.

cd AutoAvatar
export PYTHONPATH=<workspace_folder>/AutoAvatar
python exps/PosedDecKNN_dPoses_dHs/implicit_eval_dfaust.py --ws_dir <workspace_folder> --ckpt_dir <checkpoint_folder>

Pretrained Model

Download pretrained model for DFaust subject 50002 from link.

Publication

If you find our code or paper useful, please consider citing:

@inproceedings{bai2022autoavatar,
  title={AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling},
  author={Bai, Ziqian and Bagautdinov, Timur and Romero, Javier and Zollh{\"o}fer, Michael and Tan, Ping and Saito, Shunsuke},
  booktitle={European conference on computer vision},
  year={2022},
}

License

CC-BY-NC 4.0. See the LICENSE file.

facebookresearch / AutoAvatar

readme