search

creativeIKEP / HolisticMotionCapture

HolisticMotionCapture is an application and package that can capture the motion of a person with only a monocular color camera and move the VRM avatar's pose, face, and hands.
Apache License 2.0
143 stars 17 forks source link

where is mediapipe code? #2

Open lucasjinreal opened 1 year ago

creativeIKEP commented 1 year ago

@jinfagang This app uses HolisticBarracuda which has a similar implementation to official MediaPipe.

lucasjinreal commented 1 year ago

@creativeIKEP Hello, may I ask how did you get the rotations from mediapipe 3d pose landmark outpiut?

creativeIKEP commented 1 year ago

@jinfagang Each landmarks of MediaPipe is associated with each joint of humanoid, and the rotation is calculated by using the direction vector from the parent landmark to the child landmark and Quaternion.LookRotation method.

Detail implementation > HolisticMotionCapture/Packages/HolisticMotionCapture/Scripts/HolisticMotionCapture_Pose.cs

lucasjinreal commented 1 year ago

@creativeIKEP thanks, is that possible to do it in pure c++ rather than in a game engine like unity?

the 3d world landmarks are coordinates, the distance between 2 joint might be not same as bone length, does it effect? Where to get such a Humanoid skeleton tough

creativeIKEP commented 1 year ago

@jinfagang I think that pure C++ can also calculate quaternions from MediaPipe landmarks. HolisticMotionCapture calculates the quaternion of each joint of the avatar based on FK. In other words, HolisticMotionCapture only calculating which direction to rotate the joints, so I don't think the length of the avatar's bones matters unless you need to be able to reach certain places like IK.