search

Inferencer / LipSick

🤢 LipSick: Fast, High Quality, Low Resource Lipsync Tool 🤮
The Unlicense
185 stars 28 forks source link

Example of upcoming box removal feature #17

Closed Inferencer closed 6 months ago

Inferencer commented 6 months ago

Just though I would share an example of the upcoming much needed face mask/ box removal feature, bear in mind custom frames are not used on this video so the skin quality is a bit off, it runs super fast but unfortunately to achieve this effect there is tracking of the lipsynced face which is going to slow things down, we can't re-use the tracking data from the original video due to chin's moving while speaking. right now I'm just adding features but will make everything more suitable for people who are offering this tool for SaaS so data is reused etc. I assume this will be implemented next week and I will close this issue when it is

https://youtu.be/x3Ufuef1nEE

kowalgregy commented 6 months ago

Looks promising! Great work!

danablend commented 6 months ago

This is amazing!

jakebloom commented 6 months ago

Thanks so much - looking forward to this!

skeletonNN commented 6 months ago

Maybe Face Parsing can solve this bbox removal!

Inferencer commented 6 months ago

Maybe Face Parsing can solve this bbox removal!

Yeah that's a good approach I will look into Real-time adaptive skin detection as that seems promising,

I already set up landmark based alpha masking with a convex hull shape last week on my local PC then did a day of bug fixing and spent the last few days changing the file structures I keep thinking I'm nearly done but one issue solved leads to another.

It's currently setup to use the landmarks 3-15 as the shape but when it's uploaded we can play around with it. I've included a landmark map for reference.

1592469192-74364

Inferencer commented 6 months ago

Feature added to the main branch

univers3 commented 6 months ago

Is the code on HF updated too?

Inferencer commented 6 months ago

Is the code on HF updated too?

Not yet I'm afraid, it's because some file saving paths have changed so things are saved in folders then deleted but not sure HF will allow that so might need to be changed back to gradio temp folders which means I would need to detect the paths it's being saved to.

jitendra-koodo commented 5 months ago

Looking forward to this..