search

jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema
https://clapper.app
GNU General Public License v3.0
2.08k stars 194 forks source link

Add a speech-to-video workflow for MuseTalk (hosted on HF) #111

Open jbilcke-hf opened 2 months ago

jbilcke-hf commented 2 months ago

MuseTalk is a model that was published some months ago (see https://huggingface.co/spaces/TMElyralab/MuseTalk) but it is not easy to use from an API.

I need to work on the automatic hugging face spawner (to automatically deploy and bill spaces) but until I have time to do this, I think we can just reserve the workflow for private use, by doing this:

  1. add a workflow for MuseTalk (I will use my private fork https://huggingface.co/spaces/jbilcke-hf/MuseTalk)
  2. add an environment variables for allow/reject lists of workflow IDs
  3. make sure we properly hide the MuseTalk workflow if it is in the rejection list
jbilcke-hf commented 2 months ago

Well actually there is also a workflow on Replicate, it doesn't seem very fast but I've added it anyway

For the demo I still need the HF-hosted MuseTalk though, on a large GPU