Lipsync using blob instead of mp3 file

[eusthace811]

Hi,

I can see that you are going to soon merge and add the option for Lipsync using audio file mp3.

Is there a way that we could add Lipsync using blob data instead of a file?

Thanks and great lib!

[guansss]

Hi, thanks for your interest in the library!

To play an audio blob, you need to generate a URL form it. There are two ways I can think of:

1. Use the URL.createObjectURL() method.
2. Convert the blob to a base64 string and use the data: scheme to create a URL, like data:audio/mp3;base64,....

After you have the URL, you can pass it to the speak() method like you would with a file URL.

[eusthace811]

Thanks a lot for your help. Not sure if I know how to convert the blob to a base64 string, but I will try to find how.

Also, Is there a way to know when the audio ended and is not playing anymore?

Something like:

model.addEventListener("ended", () => {

});

[guansss]

Yes, basically you can access the audio element right after starting a lip sync:

const success = await model.speak(someUrl)

if (success) {
  model.internalModel.lipSync.currentAudio.addEventListener("ended", () => {})
}

currentAudio only exists during playback, and will become undefined when the ended event is fired.

[RaSan147]

Thanks a lot for your help. Not sure if I know how to convert the blob to a base64 string, but I will try to find how.

It also supports all url, blob and b64, so you can pick your choice and use here. Happy coding