search

TMElyralab / MuseTalk

MuseTalk: Real-Time High Quality Lip Synchorization with Latent Space Inpainting
Other
2.87k stars 357 forks source link

High-quality zero-shot lipsync pipeline built on MuseTalk #195

Open mvoodarla opened 2 months ago

mvoodarla commented 2 months ago

Hey folks! My team has been exploring zero-shot lipsyncing for a bit and we think we've improved on MuseTalk's quality quite a bit by using LivePortrait to neutralize expression and CodeFormer to enhance. Here's an example.

https://github.com/user-attachments/assets/cfabcd9f-92e0-4c52-b786-77fc63eef81b

We wrote a technical blog on it: https://www.sievedata.com/blog/sievesync-zero-shot-lipsync-api-developers

Hope to put out an OSS repo soon too :)

Anything we don't talk about in the blog that we should in our repo release?

eoffermann commented 2 months ago

I like this - using CodeFormer to do cleanup as a late step in the process is great. I'm doing something right now that leverages MuseTalk as part of a content automation platform and having used all of the packages you mention, from MediaPipe through to CodeFormer, this feels like a really intuitive solution. Thanks for sharing!

mvoodarla commented 2 months ago

Of course!

mvoodarla commented 1 month ago

we just put out a repo here! https://github.com/sieve-community/sievesync

iv2985 commented 1 month ago

To anyone curious, they are trying to sell a service API for $0.50/min...

min-star commented 1 month ago

but there still have teeth problem!the teeth is vague!