search

robot-perception-group / AirPose

This repository contains the code of AirPose, our multi-view fusion network for Human Pose and Shape Estimation method
MIT License
52 stars 6 forks source link
deep-learning drones human multi-view pose-estimation python shape-estimation

readme

AirPose

AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation

iccv_teaser

Check the teaser video

This repository contains the code of AirPose a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerialvehicles (UAVs) with on-board RGB cameras and computation.

Please clone the repository with the following

git clone https://github.com/robot-perception-group/AirPose.git --recursive

Data can be freely accessed here. Please download the data, and untar it whenever necessary. Content details are following:

The code was tested using Python 3.8.

SMPLX submodule in this repo is a modified version of the official SMLX implementation. Download the SMPLX model weights from here and run the following

# from the download location
unzip models_smplx_v1_1.zip -d models_smplx
unzip models_smplx/models/smplx/smplx_npz.zip -d used_models
rm models_smplx -r

Then copy the content of used_models (just created, with SMPLX_{MALE,FEMALE,NEUTRAL}.npz files) folder into your_path/AirPose/copenet/src/copenet/data/smplx/models/smplx.

You need to register before being able to download the weights.

Now, you may want to create a virtual environment. Please be sure your pip is updated.

Install the necessary requirements with pip install -r requirements.txt. If you don't have a cuda compatible device, change the device to cpu in copenet_real/src/copenet_real/config.py and copenet/src/copenet/config.py.

In those files (copenet_real/src/copenet_real/config.py and copenet/src/copenet/config.py) change LOCAL_DATA_DIR to /global/path/AirPose/copenet/src/copenet/data".

Check out this link to fix the runtime error RuntimeError: Subtraction, the-operator, with a bool tensor is not supported due to the Torchgeometry package.

Install the copenet and copenet_real packages in this repo

pip install -e copenet
pip install -e copenet_real

Download the head and hands indices files form here and place them in AirPose/copenet/src/copenet/data/smplx (MANO_SMPLX_vertex_ids.pkl and SMPL-X__FLAME_vertex_ids.npy).

Synthetic data training

The data to be used is copenet_synthetic_data.tar.gz (here)

Preprocess

To run the code of this repository you first need to preprocess the data using

# from AirPose folder
python copenet/src/copenet/scripts/prepare_aerialpeople_dataset.py /absolute/path/copenet_synthetic

And code can be run by the following (from AirPose/copenet folder):

python src/copenet/copenet_trainer.py --name=test_name --version=test_version --model=muhmr --datapath=/absolute/path/copenet_synthetic --log_dir=path/location/ --copenet_home=/absolute/path/AirPose/copenet --optional-params...

The datapath is the location of the training data.

--model specify the model type between [hmr, muhmr, copnet_singleview, copenet_twoview] which corresponds to the Baseline, Baseline+multi-view, Baseline+Fullcam and AirPose respectively.

Logs will be saved in $log_dir/$name/$version/

optional-params is to be substituted with the copenet_trainer available params as weights, lr..

Evaluation on the synthetic data

For model type [muhmr, copenet_twoview].

cd AirPose/copenet_real

python src/copenet_real/scripts/copenet_synth_res_compile.py "model type" "checkpoint Path" "/path to the dataset"

For model type [hmr, copenet_singleview], the provided checkpoint is trained with an older pytorch lightning version (<=1.2). If you want to use them, install pytorch-lightning<=1.2.

We provide the precalculated outputs on the syntehtic data using these checkpoints.

To generate the metrics, run

cd AirPose/copenet_real

python src/copenet_real/scripts/hmr_synth_res_compile.py "model type" "precalculated results directory Path" "/path to the dataset" "your_path/AirPose/copenet/src/copenet/data/smplx/models/smplx"

Fine-tuning on real dataset

The data to be used is copenet_dji_real_data.tar.gz(here).

Install the human body prior from here and download its pretrained weights (version 2) from here. Set the vposer_weights variable in the .../AirPose/copenet_real/src/copenet_real/config.py file to the absolute path of the downloaded weights (e.g. /home/user/Downloads/V02_05). If you do NOT have a GPU please change human_body_prior/tools/model_loader.py line 68 from state_dict = torch.load(trained_weigths_fname)['state_dict'] to state_dict = torch.load(trained_weigths_fname, map_location=torch.device('cpu'))['state_dict']

Note: for the hmr (Baseline) model pytorch-lightning<=1.2 is required. You might have to recheck requirements, or reinstall the requirements you can find in the main folder of this repo.

Code can be run by the following (from AirPose/copenet_real/ folder)

python src/copenet_real/copenet_trainer.py --name=test_name --version=test_version --model=hmr --datapath=path/location --log_dir=path/location/ --resume_from_checkpoint=/path/to/checkpoint --copenet_home=/absolute/path/AirPose/copenet --optional-params...

The datapath is the location of the training data.

--model specify the model type between [hmr, copenet_twoview] which corresponds to the Baseline, AirPose respectively.

The --resume_from_checkpoint is path to the pretrained checkpoint on the synthetic data.

Evaluation on real data

Install graphviz dependency with pip install graphviz in the same virtual environment.

Following code will generate the plots comparing the results of the baseline method, AirPose and AirPose+ on the real data.

This can be run from AirPose folder.

python copenet_real_data/scripts/bundle_adj.py "path_to_the_real_dataset" \\
"path_to_the_SMPLX_neutral_npz_file" \\
"path_to_vposer_folder" \\
"path_to_the_hmr_checkpoint_directory" \\
"path_to_the_airpose_precalculated_res_on_realdata_pkl" \\
"path_to_the_SMPLX_to_j14_mapping_pkl_file" \\
"type_of_data(train/test)"

Note that:

The evaluation code above needs precalculated results on the real data which are provided with the dataset. If you want to calculate them yourself, run the following code and save the variable outputs in a pkl file when a breakpoint is hit. The pkl files provided with the data are generated in the same way. For AirPose

python copenet_real/src/copenet_real/scripts/copenet_real_res_compile.py "checkpoint Path" "/path to the dataset"

For Baseline

python copenet_real/src/copenet_real/scripts/hmr_real_res_compile.py "checkpoint Path" "/path to the dataset"

Testing the client-server synchronization mechanism

To this end you need to install ros-{melodic,noetic} in your pc (Ubuntu 18.04-20.04).

Please follow the instructions that you can find here

After that you need to install the following dependencies:

sudo add-apt-repository ppa:joseluisblancoc/mrpt-stable

Navigate to your catkin_ws folder (e.g. AirPose/catkin_ws) and run:

touch src/aircap/packages/optional/basler_image_capture/Grab/CATKIN_IGNORE
touch src/aircap/packages/optional/ptgrey_image_capture/Grab/CATKIN_IGNORE

this applies to ros-melodic

Firstly, checkout the AirPose branch ros-melodic.

Be sure to update the submodule (first command).

git submodule update 
sudo apt install libmrpt-dev mrpt-apps
cd /your/path/AirPose/catkin_ws
touch src/aircap/packages/3rdparty/mrpt_bridge/CATKIN_IGNORE
touch src/aircap/packages/3rdparty/pose_cov_ops/CATKIN_IGNORE
sudo apt install -y ros-melodic-octomap-msgs ros-melodic-cv-camera ros-melodic-marker-msgs ros-melodic-mrpt-msgs ros-melodic-octomap-ros ros-melodic-mrpt-bridge ros-melodic-mrpt1

this applies to ros-noetic

sudo apt install libmrpt-poses-dev libmrpt-obs-dev libmrpt-graphs-dev libmrpt-maps-dev libmrpt-slam-dev -y
sudo apt install -y ros-noetic-octomap-msgs ros-noetic-cv-camera ros-noetic-marker-msgs ros-noetic-mrpt-msgs ros-noetic-octomap-ros ros-noetic-mrpt2

Then you can run catkin_make from the catkin_ws folder to build the whole workspace.

To run the client-server architecture you need:

To test the code you can do the following.

In separated terminals (with the workspace sourced) run:

At this point you should be able to see play the rosbag in the way you prefer.

The published topics, for each machine, are:

The ROI message can be either used as "grountruth" box with the following structure:

ymin = ymin
ymax = ymax
ycenter = xmin
xcenter = xmax

Or as a more general box where you specify the center and the height of the box. In that case a 3:4 aspect ratio is considered.

ymin = ymin
ymax = ymax
xcenter = x_center_of_the_bb
ycenter = y_center_of_the_bb

You can also create your bag and provide your own data to the tool. To that end you can check the code available here that uses a csv with the needed information (image paths, bounding boxes, and camera info) to build the bags.

Note that this is no different than running the inference manually, except for the fact that this runs at 4FPS and has the synchronization procedure enabled as explained in the paper.