AmmarkoV
commented
11 months ago The issue of inverted or broken arms bending the wrong way in its core stems from the mathematical ambiguity of regressing a 3D structure from 2D keypoints.
MocapNET receives 2D points as input and regresses a 3D BVH skeleton that corresponds to the 2D observations.
Mathematically however it is almost impossible for the method to discern e.g. the following 4 poses
This is because although the 3D orientations of the arms are different the 2D input that leads to these 3D outputs is essentially the same.
This is an issue with any 2D to 3D pose estimation method but also MNET3. To combat this issue a number of steps are taken :
1) Since the times of MNET2, I isolated a large number of corrupt files from the CMU/BVH Mocap dataset ( https://drive.google.com/file/d/1Zt-MycqhMylfBUqgmW9sLBclNNxoNGqV/view?usp=drive_link see needfix subfolder ) after manually going through all files. These data samples where corrupting training by teaching the network to bend arms backwards
2) I introduced a penalization term in the HCD fine tuning ( https://github.com/AmmarkoV/RGBDAcquisition/blob/master/opengl_acquisition_shared_library/opengl_depth_and_color_renderer/src/Library/MotionCaptureLoader/ik/bvh_inverseKinematics.c#L456 ) that is controlled by supplying --penalizeSymmetriesHeuristic as a commandline parameter, this implementation is a little shoddy and slow btw
3) While creating the training data supplying the --filterOccludedJoints flag to the GroundTruthGenerator utility filters out poses with the hands behind the torso thus improving this issue
https://github.com/AmmarkoV/RGBDAcquisition/blob/master/opengl_acquisition_shared_library/opengl_depth_and_color_renderer/src/Library/MotionCaptureLoader/export/bvh_export.c#L123
The particular dataset used to work ok, but it seems some new optimizations done in the HCD module (for MNET4) now inversely affect MNET3 return solutions with incorrect arm orientations.
I know that the cause is the HCD module and not the neural network since by disabling the IK module with the --noik command
./MocapNET2CSV --from con0014/2dJoints_v1.4.csv --noik --mt --show 3 --hands --label s10-p01-c0014-f --seriallength 4 --gdpr --penalizeSymmetriesHeuristic as you will see just taking the input from the neural network hands are bending forward..
TLDR: Due to changes in the Fine tuning module for MNET4 I seems to have broken it in MNET3 Please consider using this version : https://colab.research.google.com/github/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/mocapnet4.ipynb Clicking the first, third, fifth and sixth play button you will be able to successfully run the same dataset and recover the BVH file which looks like this.
Sorry about the inconvenience, maintaining the same HCD module for 4 different branches (out of which the MNET4 is written in Python and has another wrapper) is a nightmare, however I will do my best to resolve this.
Looking forward for comments or "better" workarounds in the case you can help with this. The "offending" code is in MocapNET/dependencies/RGBDAcquisition/opengl_acquisition_shared_library/opengl_depth_and_color_renderer/src/Library/MotionCaptureLoader/ik/
justinjohn0306
ArEnSc
notbugnotwork
It works now but the output joints from the bvh file is messed up, as you can see in the image below
Originally posted by @justinjohn0306 and @ArEnSc in https://github.com/FORTH-ModelBasedTracker/MocapNET/issues/115#issuecomment-1848873900