search

natlamir / PiperUI

A UI for the Piper TTS
40 stars 7 forks source link

Realtime Talking-Head Lipsync #5

Open sarutobiumon opened 7 months ago

sarutobiumon commented 7 months ago

@natlamir Looks like several people have already managed to speed up the time needed to generate a lip-sync'ed animated image to 8 times faster than wav2lip, in near real time speeds and has quietly released his updated code on github below:

https://github.com/yerfor/GeneFace

There is also another one that allows you to manipulate the face movement in real time (500 times faster than wav2lip): https://github.com/ashawkey/RAD-NeRF https://www.youtube.com/watch?v=JUqnLN6Q4B0

Another realtime lip-sync project that only uses photos (demo inside): https://github.com/pnkvalavala/digitaltwin

Here is a working demo built with Azure Avatar using an animated image that keeps looping in an idle "listening" state: https://www.youtube.com/watch?v=4dc4gr10ejo

Also found another super interesting project that adds styles to the generated talking faces: https://github.com/FuxiVirtualHuman/styletalk

Hope to see this code on one of your awesome WebUI projects! Thanks so much!

cjw-d commented 7 months ago

I'm wondering which project can generate lip-sync in real time speed for live.Could you please give some suggestions?Thanks!

natlamir commented 6 months ago

Thanks, I will check those out. Would like to test out which method provides both high accuracy and also fast speed.