Active3DPose

This code supplements the following paper:

Proactive Multi-Camera Collaboration for 3D Human Pose Estimation (ICLR 2023)

Project Site (Demo) | Getting Started | Training with RLlib | Evaluation & Visualization | Citation | License

TL;DR

• We simulate human crowds and aerial cameras with UE4 (currently support 4 different environment settings).
• We use Ray RLlib to train the camera agents. The RL agents learn to collaborate with each other to estimate 3D human pose in a multi-camera setting.
• We build a dedicated tool to evaluate and visualize the performance of the camera agents.

Demo

SchoolGym: 5 Cameras, 6 Humans	UrbanStreet: 5 Cameras, 6 Humans

Getting Started

Project Directory

${PROJECT_ROOT}
├── activepose
├── binary
│   ├── < Need to download from external source >
│   └── ...
├── checkpoints
│   ├── < Need to download from external source >
│   └── ...
├── configs
├── experiments
├── run
├── README.md
├── train.py
...

Preparation

1.Make sure you have the library gdown installed.

pip install gdown

cd {PROJECT_ROOT}

2. Under {PROJECT_ROOT}, enter to following command to download the pre-trained 2D pose estimation model checkpoint:

   gdown 1_WGlsM0KP5dK3YFILsbvwX4NQ60pn4XC
   unzip checkpoints.zip && rm checkpoints.zip

3. Download our UE4 Linux binaries and put them under {PROJECT_ROOT}/binary

gdown 11AwSSxrKYA2HKo7224WDpvFPsUHRVBy7
unzip binary.zip && rm binary.zip

4. Install conda environment from conda-recipe.yaml and activate it.

conda env create -f conda-recipe.yaml
conda activate active-pose

Enter the following command in case of permission issue, running binary require writing permission to the unrealcv.ini file:

chmod -R 755 binary/ && chmod -R 755 checkpoints/

5. (OPTIONAL) We have provided the option to use TensorRT. Doing so would require the users to compile the model with Torch-TensorRT specific to the spec of their own machines. If you wish to use TensorRT engine to accelerate model inference, please follow the instructions here.

Main Dependencies

• python = 3.9
• pytorch
• ray[rllib] == 1.13.0
• unrealcv
• opencv-python
• pyqt

Please refer to conda-recipe.yaml for a complete list of python dependencies.

Training

Detailed Procedure: [Training with RLlib]

Evaluation & Visualization

Detailed Procedure: [Evaluate Policy and Visualization]

Need Helps?

Please consider submitting an GitHub issue.

Citation

@inproceedings{ci2023proactive,
  title={Proactive Multi-Camera Collaboration for 3D Human Pose Estimation},
  author={Hai Ci and Mickel Liu and Xuehai Pan and fangwei zhong and Yizhou Wang},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023},
  url={https://openreview.net/forum?id=CPIy9TWFYBG}
}

License

Active3DPose and its associated simulation environment UnrealPose are released under the Apache License, Version 2.0.