search

ButzYung / SystemAnimatorOnline

XR Animator, AI-based Full Body Motion Capture and Extended Reality (XR) solution, powered by System Animator Online
https://sao.animetheme.com/XR_Animator.html
877 stars 78 forks source link

Coordinate conversion #15

Open spencer17x opened 1 year ago

spencer17x commented 1 year ago

May I ask how to convert the coordinates obtained by mediapipe into the coordinates in the threejs scene?

ButzYung commented 1 year ago

It's basically a math issue. In short, you need to calculate the rotation on each joint from the mediapipe data, starting from the root (hip) and then down the hierarchy. Basically you get the rotation of a joint by forming an axis with 2 points (usually the joint itself and its child bone) and then compare it with the rest pose axis (usually y or x axis from the T-pose) by using function like setFromUnitVectors

spencer17x commented 1 year ago

Can you give me some information about the phase algorithm? Or is there a similar library

ButzYung commented 1 year ago

To give a simplified example, let's say you want to calculate the rotation of left elbow in three.

  1. Calculate the rotation of hip by forming a Vector3 axis consisting of left hip and right hip coordinates, compare it with the T-pose of hip (x axis) and get the quaternion angle from Quaternion.setFromUnitVectors
  2. Calculate the rotation of spine by forming a Vector3 axis consisting of the center of left right hip and the center of left right shoulder coordinates, offset the parent bone rotations (hip), compare it with the T-pose of spine (y axis) and get the quaternion angle from Quaternion.setFromUnitVectors
  3. Calculate the rotation of chest (shoulder center) by forming a Vector3 axis consisting of left shoulder and right shoulder coordinates, offset the parent bone rotations (hip+spine), compare it with the T-pose of chest (x axis) and get the quaternion angle from Quaternion.setFromUnitVectors
  4. Calculate the rotation of left upper arm by forming a Vector3 axis consisting of left shoulder and left elbow coordinates, offset the parent bone rotations (hip+spine+chest), compare it with the T-pose of upper arm (x axis) and get the quaternion angle from Quaternion.setFromUnitVectors
  5. Calculate the rotation of left elbow by forming a Vector3 axis consisting of left elbow and left hand coordinates, offset the parent bone rotations (hip+spine+chest+left arm), compare it with the T-pose of lower arm (x axis) and get the quaternion angle from Quaternion.setFromUnitVectors

Talking about existing library for similar purpose, the closest I can think of is kalidokit, but their algorithm is a bit oversimplified, and it seems their library is not maintained anymore.