search

FORTH-ModelBasedTracker / MocapNET

We present MocapNET, a real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned neural networks that regress the 3D human pose by also allowing for the decomposition of the body to an upper and lower kinematic hierarchy. This permits the recovery of the human pose even in the case of significant occlusions. (c) An efficient Inverse Kinematics solver that refines the neural-network-based solution providing 3D human pose estimations that are consistent with the limb sizes of a target person (if known). All the above yield a 33% accuracy improvement on the Human 3.6 Million (H3.6M) dataset compared to the baseline method (MocapNET) while maintaining real-time performance
https://www.youtube.com/watch?v=Jgz1MRq-I-k
Other
851 stars 136 forks source link

Tensorflow/Keras code for training on custom dataset #41

Closed whitealex95 closed 3 years ago

whitealex95 commented 3 years ago

Hi, I want to use your mocap net architecture on my custom dataset. Your paper says that you trained the model using the tensorflow/keras, but I have hard time finding out how the network architecture is actually created. Is the training part of the code (data preprocessing / ensemble / SNN network architecture / lr scheduling, etc) available? Thanks alot

AmmarkoV commented 3 years ago

Hello, due to limitations imposed by FORTH the training code is not and will not be available, we are just publishing the demo and the pretrained networks used in the paper under this license. This is not my decision and not my fault, its institute policy :(

The ensemble architecture is thoroughly explained in the papers 1 and 2

The dataset used as well as the GroundTruthGenerator utility used to create the training datasets are also included.

You can use netron to open the .pb files and take a peek in their internal organization.

Given all these "ingredients" one could make his own homebrew MocapNET..