Appropriate steps to fit SKEL to 3d pose estimation keypoints

[chags1313]

This is truly amazing work!

I'm wondering if you have any suggestions for the best methods to fit the SKEL model to 3d keypoints extracted from a pose estimation model such as mediapipe. Here is what I have tried so far with poor results.

• Mapped my pose keypoints to SMPL format.
• Orient the 3d keypoints such that the mid hip is the root [0,0,0].
• Orient the hip vector so it is aligned with the x-axis.
• Orient the torso vector so it is aligned with the y-axis.
• Scale pose keypoints by shoulder width from the SKEL model original t-pose joints.
• Run parameter optimization similar to that in fit2joints script.

I've also tried to get the axis-angles from the 3d keypoints and input them as the pose parameters, which sometimes works better (also accounting for the offset in shoulder abduction in the original pose), but still runs into issues.

I'm assuming the results may be related to the limitations of 3d pose estimation (specifically estimating the z-coordinates), but may also be impacted by my processing steps. @MarilynKeller I'm wondering if you have any suggestions or resources that may help me improve my process. Any suggestions would be amazing. Thank you!!

[MarilynKeller]

Thanks :)

When you say "Mapped my pose keypoints to SMPL format." Do you mean you get 3D SMPL joint locations, or other kind of keypoints on the body surface or so?

A visual illustration of what keypoints you want to fir SKEL to can help me better answer.

But in anycase, eventually a parameter optimization similar to that in fit2joints script is probably the way to go in your case.