search

met4citizen / TalkingHead

Talking Head (3D): A JavaScript class for real-time lip-sync using Ready Player Me full-body 3D avatars.
MIT License
350 stars 108 forks source link

Custom 3D model? #3

Closed impactcolor closed 9 months ago

impactcolor commented 9 months ago

How can I add a custom 3D model? What rigging should it have?

met4citizen commented 9 months ago

The class supports full-body Ready Player Me avatars (GLB). Here is an example that you can download:

https://models.readyplayer.me/64bfa15f0e72c63d7c3934a6.glb?morphTargets=ARKit,Oculus+Visemes,mouthOpen,mouthSmile,eyesClosed,eyesLookUp,eyesLookDown&textureSizeLimit=1024&textureFormat=png

Their rig has a Mixamo-compatible bone structure described here:

https://docs.readyplayer.me/ready-player-me/api-reference/avatars/full-body-avatars

Please note that contrary to what is mentioned on that page, the texture atlassing is off by default. Therefore, as of now, the class assumes the presence of separate meshes for 'EyeLeft', 'EyeRight', 'Wolf3D_Head', and 'Wolf3D_Teeth'.

For lip-sync and facial expressions, you also need to have ARKit and Oculus compatible blend shapes (including the additional ones) all listed here:

https://docs.readyplayer.me/ready-player-me/api-reference/avatars/morph-targets/apple-arkit https://docs.readyplayer.me/ready-player-me/api-reference/avatars/morph-targets/oculus-ovr-libsync

met4citizen commented 9 months ago

I just updated the TalkingHead class to support texture atlas, thus eliminating the need for separate meshes. I will close this issue, but feel free to ask further questions if any.