A Body Part Embedding Model with Datasets for Measuring 2D Human Motion Similarity

PyTorch implementation of A Body Part Embedding Model with Datasets for Measuring 2D Human Motion Similarity, IEEE Access.

We propose a learning framework with datasets to generate embeddings for measuring motion similarity for each human body part. See project page for the details.

Getting Started

Installation

Clone this repo

git clone https://github.com/chico2121/bpe.git
cd bpe

Install dependencies
```
pip install -r requirements.txt
```
Working directory is assumed to be bpe (or bpe-master). Make sure to export PYTHONPATH:
```
export PYTHONPATH=$PYTHONPATH:/absolute-path-to/[bpe or bpe-master]
```

Data preparation

Download and place SARA dataset in bpe-datasets. Please follow the instructions to download and prepare the dataset.
Download and place NTU motion similarity annotations in bpe-datasets.

Trainining

Train the model on GPU:
```
python train_bpe.py
-g 0
--data_dir <path_to>/SARA_released
--use_footvel_loss
--logdir <path_to_dir>
```
Execute python train_bpe.py --help for parameters description:

-data_dir : Path to the data

—use_footvel_loss: To use foot velocity loss.

-logdir: Path to logdir where tensorboard visualization and model parameters are saved

Inference

Measuring correlation score between predicted similarity and AMT annotations of NTU:

python python bin/ntu_similarity.py
--data_dir <path to>/SARA_released
--model_path <path to directory with model checkpoints (e.g. exp-bpe/model/*.pth) >
--ntu_dir <path to pose data of ntu (i.e. NTU_motion_sim_annotations/refined_skeleton) >

Execute python bin/ntu_similarity.py --help for more options.

Inference code producing video output for motion similarity for a pair of videos:
```
python python bin/inference_single_pair_visuals.py
--data_dir <path to data dir>
--model_path <path to model checkpoint>
--video1 <path to video1's pose sequence>
--video2 <path to video2's pose sequence>
-h1 <video1's height>
-h2 <video2's height>
-w1 <video1's width>
-w2 <video2's width>
--use_flipped_motion
--video_sampling_window_size <number of frames>
--video_sampling_stride <stride in number of frames>
--similarity_measurement_window_size <number of oversampled video sequences>
--out_filename <filename>.mp4
--thresh 0.4
```
Execute python bin/inference_single_pair_visuals.py --help for parameters description:

--data_dir : Path to the data (include meanpose, stdpose npy files)

--model_path : Path to the model checkpoint

-video1, -video2 : JSON path of the pose sequences of video 1 and video 2

-h1, -h2 : Image height of video 1 and video 2

-w1, -w2 : Image width of video 1 and video 2

-use_flipped_motion: Using flipped body skeleton when measuring similarity

--video_sampling_window_size : parameter for sliding window sampling

--video_sampling_stride : stride of sliding window

--similarity_measurement_window_size : number of sampled sequences used for similarity score averaging

--out_filename : path and name of the output video file

--thresh : threshold to visualize body parts where similarity score is above or below the thresh([-1, 1])

Citation

If you use this code for your research, please cite the paper:

@ARTICLE{9366759,
  author={J. {Park} and S. {Cho} and D. {Kim} and O. {Bailo} and H. {Park} and S. {Hong} and J. {Park}},
  journal={IEEE Access}, 
  title={A Body Part Embedding Model With Datasets for Measuring 2D Human Motion Similarity}, 
  year={2021},
  volume={9},
  number={},
  pages={36547-36558},
  doi={10.1109/ACCESS.2021.3063302}}

Acknowledgments

This code borrows heavily from 2D-Motion-Retargeting.

chico2121 / bpe

readme