met4citizen / TalkingHead

Talking Head (3D): A JavaScript class for real-time lip-sync using Ready Player Me full-body 3D avatars.
MIT License
349 stars 107 forks source link

Pose transitions #42

Closed branaway closed 5 months ago

branaway commented 6 months ago

Hi, I was reading the code and how the Pose transition works was not immediately obvious to me. The poseAvatar object seems to pick up all the bone attributes in each animation frame, but how are the values applied to the all the joints?

met4citizen commented 6 months ago

The basic idea is that there is a base pose poseBase and a target pose poseTarget. Both include sets of bone rotations (quaternions). In the animation loop, the base quaternions are updated gradually toward the target quaternions by using spherical linear interpolation with a sigmoid easing function. This is done in the method updatePoseBase. Additionally, there are delta values in poseDelta, which could be due to head rotations, breathing etc. The actual pose you see, poseAvatar, is calculated by multiplying the quaternions of the base and delta rotations in the method updatePoseDelta. The properties in the poseAvatar directly reference the model's bone quaternions.

branaway commented 6 months ago

Thanks for responding. But where is the part for " The properties in the poseAvatar directly reference the model's bone quaternions."? I see the poseAvatar makes copy of the quaternions of the joints. Would you care to quote the relevant lines?

met4citizen commented 6 months ago

After loading a new avatar in the showAvatar method, the poseAvatar properties are set to reference the avatar's bones. Here are the relevant lines:

const o = this.armature.getObjectByName(ids[0]);
this.poseAvatar.props[x] = o[ids[1]];

The copy method in quaternion merely copies the x, y, z, and w properties of the argument to the referenced object. See https://threejs.org/docs/#api/en/math/Quaternion.copy