search

FORTH-ModelBasedTracker / MocapNET

We present MocapNET, a real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned neural networks that regress the 3D human pose by also allowing for the decomposition of the body to an upper and lower kinematic hierarchy. This permits the recovery of the human pose even in the case of significant occlusions. (c) An efficient Inverse Kinematics solver that refines the neural-network-based solution providing 3D human pose estimations that are consistent with the limb sizes of a target person (if known). All the above yield a 33% accuracy improvement on the Human 3.6 Million (H3.6M) dataset compared to the baseline method (MocapNET) while maintaining real-time performance
https://www.youtube.com/watch?v=Jgz1MRq-I-k
Other
822 stars 133 forks source link

csv to bvh problem #110

Open Jeckinchen opened 9 months ago

Jeckinchen commented 9 months ago

Hello, I am currently using the file "src/python/mnet4/csvNET.py". I noticed that in the file con0014/2dJoints_v1.4.csv downloaded from Google Colab, there seem to be 138 2D keypoints. The results from my 2D pose estimator are in the coco-wholebody format with 133 keypoints. How can I use csvNET.py to obtain the corresponding BVH file?

AmmarkoV commented 8 months ago

The network assumes input in the BODY25 format :

49032982-d5bfb580-f162-11e8-935d-5139c01eeb57

Because COCO does not have foot fingers, thus making it impossible to track the orientation of the foot!

keypoints_pose_18

You can either generate "foot" points close to the heel and pass them in , or you can pass them as zeros in the hope that the neural network will do fine without them.

The order and naming scheme of joints can be found here :

https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/src/python/mnet4/holisticPartNames.py#L311

Ammar

Jeckinchen commented 8 months ago

The network assumes input in the BODY25 format :

49032982-d5bfb580-f162-11e8-935d-5139c01eeb57

Because COCO does not have foot fingers, thus making it impossible to track the orientation of the foot!

keypoints_pose_18

You can either generate "foot" points close to the heel and pass them in , or you can pass them as zeros in the hope that the neural network will do fine without them.

The order and naming scheme of joints can be found here :

https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/src/python/mnet4/holisticPartNames.py#L311

Ammar

Thanks for your answer. But I still have some problems. My 2D keypoints estimation results follow the format and order specified at https://github.com/open-mmlab/mmpose/blob/537bd8e543ab463fb55120d5caaa1ae22d6aaf06/configs/_base_/datasets/coco_wholebody.py#L13C20-L13C20 with 133 keypoints. However, I noticed that csvNET.py uses 138 keypoints. How can I convert these 133 keypoints to the 138 keypoints required by the project? Looking forward to your answer. Thank you!