Computing Arm Matrix

[JpEncausse]

Hello, Thanks for your library,

I try to use a ReadyPlayer Me Avatar with MediaPipe Pose estiation. The bone's name is very close to Mixamo, so I test your avatar.js code to give a try into my messy code.

• In MediaPipe I set the selfieMode to true (I use pose Estimation that provide arms/legs, etc ...)
• The spine is ok, when bending forward, the avatar bend forward
• The arm is a real mess it seems I missed something :-(

Here is a little video: https://youtu.be/2T_EuThSKB0

I focus on that part of the code :

        let xAxis = shoulderX.clone();
        let yAxis = shoulderY.clone();
        let zAxis = shoulderZ.clone();
        let basis = new THREE.Matrix3().set(
            xAxis.x,  yAxis.x,  zAxis.x,
            xAxis.y,  yAxis.y,  zAxis.y,
            xAxis.z,  yAxis.z,  zAxis.z,
        );

        let rot = this.rotateBone(userJoints[LEFTSHOULDER], userJoints[LEFTELBOW], this.bones.leftElbowBone.position, basis);
        this.bones.leftShoulderBone.quaternion.slerp(rot, SMOOTHING);
        this.updateBasis(this.bones.leftShoulderBone.quaternion, xAxis, yAxis, zAxis, basis);

        rot = this.rotateBone(userJoints[LEFTELBOW], userJoints[LEFTWRIST], this.bones.leftWristBone.position, basis);
        this.bones.leftElbowBone.quaternion.slerp(rot, SMOOTHING);
        this.updateBasis(this.bones.leftElbowBone.quaternion, xAxis, yAxis, zAxis, basis);

        let leftFingersUser = userJoints[LEFTPINKY].lerp(userJoints[LEFTINDEX], 0.5);
        let leftFingersAvatar = this.bones.leftHandBones[PINKY1].position.clone().lerp(this.bones.leftHandBones[INDEX1].position, 0.5);
        rot = this.rotateBone(userJoints[LEFTWRIST], leftFingersUser, leftFingersAvatar, basis);
        this.bones.leftWristBone.quaternion.slerp(rot, SMOOTHING);

In my code all the bones are scoped to this.bones and the helper fuctions to this. In the results, it's like there is an error with an axis ?! How do you debug that behavior (I'm very bad with 3D stuff)

I move all the arm/leg code into a function that take the left/right side

_connect(child, srcSide, tgtSide){
    if (child.name == srcSide+'Arm')        { this.bones[tgtSide+'ShoulderBone']  = child }
    if (child.name == srcSide+'ForeArm')    { this.bones[tgtSide+'ElbowBone']     = child }
    if (child.name == srcSide+'Hand')       { this.bones[tgtSide+'WristBone']     = this.bones[tgtSide+'HandBones'][0] = child }

    if (child.name == srcSide+'UpLeg')      { this.bones[tgtSide+'HipBone']       = child }
    if (child.name == srcSide+'Leg')        { this.bones[tgtSide+'KneeBone']      = child }
    if (child.name == srcSide+'Foot')       { this.bones[tgtSide+'AnkleBone']     = child }
    if (child.name == srcSide+'Toe_End')    { this.bones[tgtSide+'FootBone']      = child }

    if (child.name == srcSide+'HandThumb1') { this.bones[tgtSide+'HandBones'][1]  = child }
    if (child.name == srcSide+'HandThumb2') { this.bones[tgtSide+'HandBones'][2]  = child }
    if (child.name == srcSide+'HandThumb3') { this.bones[tgtSide+'HandBones'][3]  = child }
    if (child.name == srcSide+'HandThumb4') { this.bones[tgtSide+'HandBones'][4]  = child }
    if (child.name == srcSide+'HandIndex1') { this.bones[tgtSide+'HandBones'][5]  = child }
    if (child.name == srcSide+'HandIndex2') { this.bones[tgtSide+'HandBones'][6]  = child }
    if (child.name == srcSide+'HandIndex3') { this.bones[tgtSide+'HandBones'][7]  = child }
    if (child.name == srcSide+'HandIndex4') { this.bones[tgtSide+'HandBones'][8]  = child }
    if (child.name == srcSide+'HandMiddle1'){ this.bones[tgtSide+'HandBones'][9]  = child }
    if (child.name == srcSide+'HandMiddle2'){ this.bones[tgtSide+'HandBones'][10] = child }
    if (child.name == srcSide+'HandMiddle3'){ this.bones[tgtSide+'HandBones'][11] = child }
    if (child.name == srcSide+'HandMiddle4'){ this.bones[tgtSide+'HandBones'][12] = child }
    if (child.name == srcSide+'HandRing1')  { this.bones[tgtSide+'HandBones'][13] = child }
    if (child.name == srcSide+'HandRing2')  { this.bones[tgtSide+'HandBones'][14] = child }
    if (child.name == srcSide+'HandRing3')  { this.bones[tgtSide+'HandBones'][15] = child }
    if (child.name == srcSide+'HandRing4')  { this.bones[tgtSide+'HandBones'][16] = child }
    if (child.name == srcSide+'HandPinky1') { this.bones[tgtSide+'HandBones'][17] = child }
    if (child.name == srcSide+'HandPinky2') { this.bones[tgtSide+'HandBones'][18] = child }
    if (child.name == srcSide+'HandPinky3') { this.bones[tgtSide+'HandBones'][19] = child }
    if (child.name == srcSide+'HandPinky4') { this.bones[tgtSide+'HandBones'][20] = child }
}

In the main fuction I simply call

this._connect(child, 'Left',  'right')
this._connect(child, 'Right', 'left')

So yes I follow your code, crossing left and right side.

[JpEncausse]

I manage to setup a JSFIddle if you have any tips to fix arms/legs, many many many thanks https://jsfiddle.net/8nvbtfgo/2/

[Neleac]

I will take a look when I get a chance. Meanwhile, please see the readyplayerme branch where I tried some code for RPM avatars, although with poor quality. I also have a more recent Unity project that uses ReadyPlayerMe avatars.

[JpEncausse]

Oh my got, I missed that branch ! Thanks I'll look into it. It seems the algo is close.

I only need arm/leg for now. For the face I'll trick with TTS.

Interesting you switch to Unity. I need to stock to the web for better compatibility. MediaPipe is really awesome

[arghideutis]

Hi, I'm interested too in animating Readyplayerme characters in THREEJS. I've tried the readyplayerme branch, I successfully got shoulder rotation to work, but the forearm are not mapped correctly. Have anybody found a solution?