Viseme not lip-syncing

[Eonasdan]

Hello,

Thanks for this project. I haven't found anything close to doing what I wanted.

I'm having trouble getting the avatar's lips to move. I have a backend process that will call Azure's TTS service. I store the audio (base64), an array a visemes, and words. That API also converts from nanoseconds to seconds since I saw your Azure example dividing the offsets by 10000. Using the code from the index.html file, I convert the JSON I get back to the words/visemes that your code is expecting (mapping, etc.)

I then call speakAudio providing a speak object. The audio plays and the subtitles work, but as I said, the avatar's lips do not move.

Here's a minimal sample. I use WebStorm's built-in serve functionality to host it locally.

Please let me know if you have any thoughts on what I can do to get this working. If you have a ko-fi or something, I'd be happy to donate.

[met4citizen]

The starting times and durations for both the words and visemes should be provided to the TalkingHead class in milliseconds. In your JSON file, the offset for the first word is 50, which seems right. Now, since the units should be the same, the offset for the first viseme of that word should be approximately the same, but it is 500000. So, all you need to do is divide the viseme offsets by 10000.

You should perhaps fix the units in your backend process, but to test this, I downloaded your code and made the following change, and it worked perfectly.

// const time = e.AudioOffset;
const time = e.AudioOffset / 10000;

By providing the example code of the issue, you made it very easy for me, so thank you. I was happy to help.

[Eonasdan]

THANK YOU!

I did wonder if this was an offset issue. You know what it was?? In the backend I had done the conversion but missed the viseme. After I had already generated my test JSON (to avoid Azure), I fixed the viseme offset, but I didn't regenerate the test JSON file.

Ugh. Feel so stupid now lol.

As one OSS dev to another, I know how important sample code is. My issue templates for my more popular repos always include a text box for sample location.

Aside: As I was trying to debug and understand what was happening, I converted the module to typescript. It's not very pretty right now, but if you're interested in what I have, I'm happy to share it.

[Eonasdan]

resolved