Preparation

• download datasets for evaluation
  • AGORA
  • BEDLAM
• download SMPL-X body models.
• download SMPL body models SMPL_FEMALE.pkl, SMPL_MALE.pkl, SMPL_NEUTRAL.pkl provided by SMPLer-X.
• download other SMPL-X dependent files: SMPLX_to_J14.pkl, MANO_SMPLX_vertex_ids.pkl, SMPL-X__FLAME_vertex_ids.npy, SMPLX_NEUTRAL.pkl provided by SMPLer-X.
• download AiOS checkpoint
• download AGORA validation set Humandata Organize them according to this datastructure:

  AiOS/
  ├── config/
  └── data
  ├── body_models
      └── smplx
  |       ├──MANO_SMPLX_vertex_ids.pkl
  |       ├──SMPL-X__FLAME_vertex_ids.npy
  |       ├──SMPLX_NEUTRAL.pkl
  |       ├──SMPLX_to_J14.pkl
  |       ├──SMPLX_NEUTRAL.npz
  |       ├──SMPLX_MALE.npz
  |       └──SMPLX_FEMALE.npz
      └── smpl
  |       ├──SMPL_FEMALE.pkl
  |       ├──SMPL_MALE.pkl
  |       └──SMPL_NEUTRAL.pkl
  ├── cache
  ├── checkpoint
  │   └── aios_checkpoint.pth
  ├── datasets
  │   ├── agora
  │   └── bedlam
  └── multihuman_data
      └── agora_validation_multi_3840_1010.npz

  Installtion

# Create a conda virtual environment and activate it.
conda create -n aios python=3.8 -y
conda activate aios

# Install PyTorch and torchvision.
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge

# Install Pytorch3D
git clone -b v0.6.1 https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install -v -e .
cd ..

# Install MMCV, build from source
git clone -b v1.6.1 https://github.com/open-mmlab/mmcv.git
cd mmcv
export MMCV_WITH_OPS=1
export FORCE_MLU=1
pip install -v -e .
cd ..

# Install other dependencies
conda install -c conda-forge ffmpeg
pip install -r requirements.txt 

# Build deformable detr
cd models/aios/ops
python setup.py build install
cd ../../..

Inference

• Place the mp4 video for inference under AiOS/demo/
• Prepare the pretrained models to be used for inference under AiOS/data/checkpoint
• Inference output will be saved in AiOS/demo/{INPUT_VIDEO}_out

cd main
sh scripts/inference.sh {INPUT_VIDEO} {OUTPUT_DIR} 

# For inferencing short_video.mp4 with output directory of demo/short_video_out
sh scripts/inference.sh short_video demo

Test

a. Make test_result dir

mkdir test_result

b. AGORA Validatoin

Run the following command and it will generate a 'predictions/' result folder which can evaluate with the agora evaluation tool

sh scripts/test_agora_val.sh data/checkpoint/aios_checkpoint.pth agora_val

b. AGORA Test Leaderboard

Run the following command and it will generate a 'predictions.zip' which can be submitted to AGORA Leaderborad

sh scripts/test_agora.sh data/checkpoint/aios_checkpoint.pth agora_test

c. BEDLAM

Run the following command and it will generate a 'predictions.zip' which can be submitted to BEDLAM Leaderborad

sh scripts/test_bedlam.sh data/checkpoint/aios_checkpoint.pth bedlam_test

Acknowledge

Some of the codes are based on MMHuman3D, ED-Pose and SMPLer-X.