the official implementation of ACM MM 2021 paper <VoteHMR:Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds>
This is a release of our paper <VoteHMR:Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds>
Authors: Guanze Liu, Yu Rong, Lu Sheng*
[arxiv]
Part of the code is inspired by DCT-ICCV and VoteNet
@inproceedings{liu2021votehmr,
title={VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds},
author={Liu, Guanze and Rong, Yu and Sheng, Lu},
booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
pages={955--964},
year={2021}
}Installation
The code is tested under the following environment
- pytorch 1.1.0
- python 3.7.10
- cuda 10.1
- pointnet2 (https://github.com/facebookresearch/votenet/tree/main/pointnet2)
Data Preparation
Download Human Models
the required SMPL models are uploaded to BaiduYUN [extract code: ame2] for download
Set up Blender
You need to download Blender and install scipy package to run the first part of the code. The provided code was tested with Blender2.78, which is shipped with its own python executable as well as distutils package. Therefore, it is sufficient to do the following:
/blenderpath/2.78/python/bin/python3.5m get-pip.py
/blenderpath/2.78/python/bin/python3.5m pip install scipy
/blenderpath/2.78/python/bin/python3.5m pip install numpy
Notice that get-pip.py
is downloaded from pip. Replace the blenderpath
with your own and set BLENDER_PATH
.
In order to read rendered depth images and segm images, the OpenEXR bindings for PYTHON is required.
Set the openexr_py2_path
in src/datasets/config.copy
as your OpenEXR path.
The Synthetic Dataset SURREAL and DFAUST can be downloaded from existing repositary.
We provide the shell script to generate partial point cloud from provided synthetic data.
You should set the corresponding data_path in the following shell scripts, including surreal_data_path
, surreal_save_path
, dfaust_save_path
, as well as the tmp_path
, output_path
in src/datasets/config.copy
sh scripts/generate_training_data.sh
Prepare Testing Data
We also provide the shell script to generate testing data sh scripts/generate_testing_data.sh
Training
We provide the shell scripts to train with VoteHMR
- If you wish to train on single gpu, run the following shell script sh scripts/train.sh
- If you wish to train on multiple gpu, run the following shell script sh scripts/train_dist.sh
Set the surreal_save_path
as the path to processed surreal data.
We provide the shell scripts for evaluation, set chechpoints as your model path dir.
sh scripts/test.sh
Fine Tuning on Real Data
If you wish to fine tune on real data, run
sh scripts/weakly_supervised.sh
Before training starts, to visualize the training results and loss curve in real-time, please run python -m visdom.server 8098
and click the URL https//localhost:8098