Add a new workflow category for lip syncing (VIDEO_LIPSYNC) - Githubissues

jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema

https://clapper.app

GNU General Public License v3.0

2.08k stars 195 forks source link

Add a new workflow category for lip syncing (VIDEO_LIPSYNC) #74

Closed jbilcke-hf closed 3 months ago

jbilcke-hf commented 3 months ago

Context

Since we have a decent way of generating a video baseline (eg. using Live Portrait directly from Clapper),

we need to take those videos and lip-sync them to the dialogue segments

Solution

Add a new workflow category for lip syncing (VIDEO_LIPSYNC)
Create at least one example workflow

I propose to run the lipsync workflow in the segment resolver (eg. after generating a video or applying a video face swap)

For instance, this can be done after this block:

https://github.com/jbilcke-hf/clapper/blob/main/packages/app/src/app/api/resolve/route.ts#L164-L203

jbilcke-hf commented 3 months ago

done