Speak Audio with PCM Buffer, no lip sync

[charlyhayoz]

Hi,

first thank you for this great work.

I have a problem with the function speakAudio, i pass a object with

let object = { audio: data.buffer, words: ['Ça', 'va', 'bien', 'grâce', 'à', 'toi'], wtimes: [93.75, 262.5, 481.25, 693.75, 862.5, 1118.75], wdurations: [87.5, 187.5, 212.5, 50, 100, 56.25] };

Audio is working well (Buffer PCM), but there is no animation and no lipsync. No error too.

I wonder if i forget something or it's a bug

[met4citizen]

Hi. I couldn't reproduce your problem. I copied the object into my own test app, and the lips moved fine. Of course, I didn't have the right audio, and I used the English lip-sync language, but the lips moved nevertheless. The problem must be somewhere else.

Since you don't provide visemes (visemes, vtimes, and vdurations) to speakAudio, it means that the class uses one of the lip-sync modules to generate the visemes for each word. There is no lip-sync module for French, so are you using the English module, some other supported language, or perhaps developing one for French?

[charlyhayoz]

You're right, the problem was the 3D model i used. It's strange because i downloaded it from Ready Player Me like wrote in the appendix ....

Do I forgot something in the export for Ready Player me ?

6633cd2ebc7a300396411cc5.zip

Here is attached the GLB file.

[met4citizen]

OK, that explains it. The attached file is missing most of the blend shapes (morph targets) needed for facial expressions and lip-sync. You need to manually add all the URL parameters listed in Appendix A when you download the GLB file.

Here is the correct download link for your avatar:

https://models.readyplayer.me/6633cd2ebc7a300396411cc5.glb?morphTargets=ARKit,Oculus+Visemes,mouthOpen,mouthSmile,eyesClosed,eyesLookUp,eyesLookDown&textureSizeLimit=1024&textureFormat=png