🚀 Real Time Pose Estimation in TensorLayer

[zsdonghao]

A discussion for real-time (multi-person) pose estimation using TensorLayer and TensorFlow

Paper List

• 🚀Pose Proposal Networks, ECCV 2018
• DensePose
• 🚀 Convolutional Pose Machines
• Stacked Hourglass Networks for Human Pose Estimation
• 🚀Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• For Model Acceleration, see https://github.com/tensorlayer/tensorlayer/issues/416

Existing code/resource

• 🚀the author of "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" provides some links in his github: ZheC/Realtime_Multi-Person_Pose_Estimation, [video], [training log]
• pyopenpose PyOpenPose
• 🚀COCO API for preparing the data - cocoapi , pip3 install pycocotools, see keras_Realtime_Multi-Person_Pose_Estimation
• tf-pose-estimation using MobileNet , video , blog🚀 , faster and other implementations
• data augmentation in cpp , see rmpe_dataset_server
• evaluate MPII result , see mpii-evluation
• openpose output format , see openpose
• Receptive Fields Calculator

Blog

• https://arvrjourney.com/human-pose-estimation-using-openpose-with-tensorflow-part-1-7dd4ca5c8027
• https://arvrjourney.com/human-pose-estimation-using-openpose-with-tensorflow-part-2-e78ab9104fc8

About Data Augmentation

• https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation/blob/master/training/setLayers.py#L457
• https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation/blob/master/training/setLayers.py#L343

Others

• MPII benchmarks http://human-pose.mpi-inf.mpg.de/#related_benchmarks
• TensorFlow lite
• 知乎 https://www.zhihu.com/question/59750782/answer/207238452
• CSDN CPM http://blog.csdn.net/shenxiaolu1984/article/details/51094959
• Use TF dataset API to deal with large dataset https://blog.csdn.net/dQCFKyQDXYm3F8rB0/article/details/79342369

[filipetrocadoferreira]

https://github.com/ildoonet/tf-pose-estimation

[zsdonghao]

I release the APIs to download or visualise MPII dataset in 1 line of code here: https://github.com/tensorlayer/tensorlayer/pull/482

import pprint
import tensorlayer as tl

img_train_list, ann_train_list, img_test_list, ann_test_list = tl.files.load_mpii_pose_dataset()

print(img_train_list[0])
pprint.pprint(ann_train_list[0])

for i in range(100):  # show 100 images
    image = tl.vis.read_image(img_train_list[i])
    tl.vis.draw_mpii_people_to_image(image, ann_train_list[i], '_temp%d.png' % i)

[zsdonghao]

Any ideas to make it run faster?

Algorithm

	OpenPose	Idea
image size	368x654 i.e. 9:16	244x244 ?
CNN	VGG-19	mobilenetV2 or others, see https://github.com/tensorlayer/tensorlayer/issues/416
CNN	Residual Squeeze	(1), (2)

Engineering

• Faster pre-processing : ❓
• Faster post-processing :
  • @filipetrocadoferreira c++/cython multiprocessing module
  • pypy vs cython
• TensorRT

[filipetrocadoferreira]

I think where you can achieve better results easier is in post-processing. At least in tf-pose-estimation that's the current bottleneck. maybe implementing a c++/cython multiprocessing module for that can be enough

[zsdonghao]

Implementing openpose in https://github.com/tensorlayer/tensorlayer/pull/765

[1icas]

I'm trying to implement a real time pose estimation(mulit-person) on mobile. For the mobile performence, I'm trying to quantize the model.

[zsdonghao]

@1icas I think mobilenet is a easier approach than quantisation..

[1icas]

@zsdonghao yes,i know. But it's not satisfy the performance requirement. Also i'm very interested in quantisation.

[1icas]

Or do you have other suggesstion for model acceleration ?