Algorithms to produce wrappers for

[peabody124]

This issue is to maintain a list of some algorithms that seem particularly useful to implement. Currently up on the docket are:

• [x] Deep High-Resolution Representation Learning for Human Pose Estimation https://github.com/HRNet/deep-high-resolution-net.pytorch (or could use implementation from mmpose, which has the benefit of some newer tricks like DARK and AIDP)
• [x] 3D human pose estimation in video with temporal convolutions and semi-supervised training https://github.com/facebookresearch/VideoPose3D does a pretty good job of directly producing 3D joint locations from a temporal sequence of 2D keypoints. Also, some related successors that might be good alternatives:
  • Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos https://github.com/3dpose/GnTCN.
  • implemented A Graph Attention Spatio-temporal Convolutional Network for 3D Human Pose Estimation in Video https://github.com/fabro66/GAST-Net-3DPoseEstimation
  • Several transformer based approaches recently came out. 3D Human Pose Estimation with Spatial and Temporal Transformers in particular has code available https://github.com/zczcwh/PoseFormer
  • PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillation is a bit more distant, and outputs SMPL parameters. no code, though.
  • 3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training also has good results through a temporal consistency adversarial prior. no code, though.
• [x] 3D Human Motion Estimation via Motion Compression and Refinement https://github.com/ZhengyiLuo/MEVA performs compression to a latent state space that includes the dynamic changes occurring. Results seem to outperform VIBE.
  • Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation https://github.com/JDAI-CV/DSD-SATN also implements dynamic smoothing on models, but I don't think performs as well anymore.
• [ ] PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos Code isn't released, but this approach seems particularly fruitful for improving model based estimates with 3D keypoint locations
  • HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation also has an elegant approach to mesh refinement, although it isn't clear how easily it would integrate into my framework https://github.com/Jeff-sjtu/HybrIK.
• [x] TransTrack: Multiple-Object Tracking with Transformer https://github.com/PeizeSun/TransTrack

Single object tracking

• [ ] Try out MMTrack STARK implementation

[peabody124]

Use MMPose for the HRNet implementations, which has a beautiful API and is very actively maintained. It might be worth replacing the default multi person bottom up detector (for face blurring) and the tracker with MMTracking implementations. This could remove a "default" dependency on OpenPose.

I implemented a wrapper for GAST-Net-3D. This also produces nice results, but it might be worth introducing one for VideoPose3D as well - in particularly because this produces absolute spatial coordinates too, and not just root-relative ones.