Exploiting temporal context for 3D human pose estimation in the wild uses temporal information from videos to correct errors in single-image 3D pose estimation. In this repository, we provide results from applying this algorithm on the Kinetics-400 dataset. Note that this is not an exhaustive labeling: at most one person is labeled per frame, and frames which the algorithm has identified as outliers are not labeled.
The archive contains a single .pkl file for each video where bundle adjustment succeeded. Let N be the number of frames that the algorithm considers inliers. Then the .pkl file contains a map with the following keys:
time: Array of size N, where each element is the time in seconds since the start of the 10-second kinetics clip (not the start of the whole video)smpl_shape: Array of size Nx10, where each row is the SMPL shape for one example.smpl_pose: Array of size Nx72, where each row is the SMPL pose for one example.3d_keypoints: Array of size Nx24x3 where each slice is the 19 cocoplus joints obtained from the SMPL model using the custom keypoint regressor described below.2d_keypoints: Array of size Nx19x2, where each slice is the 19 cocoplus joints reprojected from the SMPL model, using the custom keypoint regressor described below, in (x,y) coordinates. These coordinates are normalized to the image frame: therefore, (0, 0) and (1,1) are the top-left and bottom-right corners respectively.cameras: Array of size Nx3, containing the translation and scale that maps the SMPL 3D joint locations to 2d_keypoints. cameras[:,0] is scale and cameras[:,1:3] is translation. Thus, if x is a 19x3 array of 3D keypoints in the format (x,y,z) produced byt the SMPL model, then 2d_keypoints can be computed as cameras[:,0:1]*(x[:,0:2]+cameras[:,1:3]).vertices: Array of size Nx6890x3. These are the vertices of the SMPL mesh computed from smpl_shape and smpl_pose computing with the neutral body model from HMR.The dataset can be downloaded here (325 GB), as well as an significantly smaller archive which does not contain vertices, but is otherwise identical, here (2.7 GB).
We also have a custom joint regressor that is specific to our pose estimator (since there are slight differences between the 2D joints we used for bundle adjustment and those used for SMPL). This is a 6890x19 array that can be used as a drop-in replacement for the cocoplus_regressor that is distributed in the public HMR repository, and is required to extract the 3d_keypoints above from the estimated poses. It was learned using ground-truth from the Human3.6m dataset.
This Tensorflow checkpoint was trained using the procedure outlined in our paper. That is, it uses the above dataset as well as standard HMR 3D data. The checkpoint is compatible with HMR.
youtube-dl and ffmpeg to download the Kinetics videos to visualise.wget https://github.com/akanazawa/hmr/raw/master/src/tf_smpl/smpl_faces.npythird_party/activity_net. This can be done with: wget https://raw.githubusercontent.com/activitynet/ActivityNet/master/Crawler/Kinetics/download.py -P third_party/activity_net. We tested with commit 530ac3a of the download script.requirements.txt. We recommend creating a new virtual environment, and running pip install -r requirements.txt.To run the demo:
python run_visualise --filename <path_to_downloaded_pickle_file>
If you use this data, please cite
@InProceedings{Arnab_CVPR_2019,
author = {Arnab, Anurag* and
Doersch, Carl* and
Zisserman, Andrew},
title = {Exploiting temporal context for 3D human pose estimation in the wild},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}