search

RaSan147 / pixi-live2d-display

A PixiJS plugin to display Live2D models of any kind (With lip-sync from audio)
https://guansss.github.io/pixi-live2d-display/
MIT License
75 stars 16 forks source link

Sound With Subtitles #2

Open idiomer opened 1 year ago

idiomer commented 1 year ago

I'm looking for the solution: How to add subtitles with keeping them in sync with the audio? Can you give me some help. Thanks

RaSan147 commented 1 year ago

I'll try on creating some algorithms. Although audio and subtitle sync is totally different field, I'll try learning frequency analizer to match with voice output (there's also an issue, if you set volume 0 (or change volume in runtime via api), frequency can mess up (provided via browser)

RaSan147 commented 5 days ago

Okk I worked on suntitle. With proper starting and ending time and async function, you can use those timestamps to predict the word timing (google tts api v1beta allows the starting time for the words)

Then using on_start event (i guess) you can add subtitles dynamically in the text box.

Let me know if you need any help.