search

FORTH-ModelBasedTracker / MocapNET

We present MocapNET, a real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned neural networks that regress the 3D human pose by also allowing for the decomposition of the body to an upper and lower kinematic hierarchy. This permits the recovery of the human pose even in the case of significant occlusions. (c) An efficient Inverse Kinematics solver that refines the neural-network-based solution providing 3D human pose estimations that are consistent with the limb sizes of a target person (if known). All the above yield a 33% accuracy improvement on the Human 3.6 Million (H3.6M) dataset compared to the baseline method (MocapNET) while maintaining real-time performance
https://www.youtube.com/watch?v=Jgz1MRq-I-k
Other
858 stars 137 forks source link

having issues with the mnet3 branch #115

Closed justinjohn0306 closed 11 months ago

justinjohn0306 commented 11 months ago

Here's a screenshot of the error i'm facing after the initialization of the mnet3 branch of the repo. The cmd line I used for the test: ./MocapNET2CSV --from con0014/2dJoints_v1.4.csv --ik 0.001 15 30 --mt --show 3 --hands --label s10-p01-c0014-f --seriallength 4 --gdpr

Screenshot from 2023-12-08 11-31-34

AmmarkoV commented 11 months ago

Unfortunately the error looks like something has internally changed in the way tensorflow restores and executes saved model signatures (?).

My MNET3 code relies on tensorflow CPP bindings : https://www.tensorflow.org/api_docs/cc using this wrapper : https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/master/src/Tensorflow2/tensorflow2.h

The MNET3 training data was produced in 2021 with TF 2.3.1 so this might mean that tensorflow has broken compatibility https://www.tensorflow.org/guide/versions ?

If that is the case the fix would be to : 1) dump the old models to a different format e.g. ONNX and add a wrapper for ONNX execution 2) Retrain the models from scratch and save them using the new convention, however these would be new models 3) somehow Upgrade TF2.3 models to the new version ?

The best I can offer at the time is the python MNET4 backend that has body + hands + face.. :( https://colab.research.google.com/github/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/mocapnet4.ipynb

Unfortunately if this is a compatibility problem it is hard to fix

AmmarkoV commented 11 months ago

I think there is a problem with the Google Drive model files..! I reverted back to the ammar.gr version of the files and I think it fixes the issue, please confirm!

but also keep in mind the new version : https://colab.research.google.com/github/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/mocapnet4.ipynb

AmmarkoV commented 11 months ago

@justinjohn0306 I have also updated the Google Drive files and they work once again. Will keep the issue open until you confirm!

justinjohn0306 commented 11 months ago

@justinjohn0306 I have also updated the Google Drive files and they work once again. Will keep the issue open until you confirm!

It works now but the output joints from the bvh file is messed up, as you can see in the image below

Screenshot from 2023-12-10 12-06-47

justinjohn0306 commented 11 months ago

I think there is a problem with the Google Drive model files..! I reverted back to the ammar.gr version of the files and I think it fixes the issue, please confirm!

but also keep in mind the new version : https://colab.research.google.com/github/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/mocapnet4.ipynb

I did test the mnet4 branch and it wasn't accurate with the pose estimation with the mediapipe module

ArEnSc commented 11 months ago

we are running into the same issues as @justinjohn0306 here

ArEnSc commented 11 months ago

@AmmarkoV

AmmarkoV commented 11 months ago

Since the issue with model package is resolved, and the inverted arms is another issue I am opening a new issue ticket ( https://github.com/FORTH-ModelBasedTracker/MocapNET/issues/116 ) and closing this as resolved