jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema
https://clapper.app
GNU General Public License v3.0
2.06k stars 191 forks source link

Improve ComfyUI integration #5

Open jbilcke-hf opened 4 months ago

jbilcke-hf commented 4 months ago

Context

ComfyUI is all the rage right now, and I would like to use it as one of the supported backends for Clapper.

Currently, Clapper only has ultra-basic support for ComfyUI cloud providers, such as Comfy.icu (see this screenshot, the ugly text field at the very bottom):

Screenshot 2024-07-25 at 20 52 32

Goal

At minimum, it should be able to use an existing integration running locally or in the cloud

But what I have in mind is more something like this:

Limitations / things to checks

I'm not a ComfyUI developer (or user), so it's gonna be a fun ticket!

How to implement

For now I've only added some basic code that doesn't really work (search for "comfy" and "ComfyIcu" - it's related to comfy.icu - in the codebase)

Acceptance criteria

Being able to run any ComfyUI workflow, for : image, video, and why no also sound, voice and music

lalalune commented 4 months ago

/boss $250

boss-bounty[bot] commented 4 months ago

Total Bounty: $1050

Bounties

Bounties paid automatically on close:

sendn0des commented 3 months ago

Flexibility to pull from openart.ai would present some wonk. There's too many workflows and many are meh - but it could lead to an issues with frontend design later attempting to accommodate.

I'll bring over some workflows I've considered or have that would play well and can design toolkit around them.

lalalune commented 3 months ago

/boss $500

jbilcke-hf commented 3 months ago

I'm making some changes to the codebase to prepare and facilitate work for this ticket.

These changes are not specific to ComfyUI, so not in the scope of this ticket, and not meant to claim the bounty:

  1. I've added the concept of workflow to Clapper and the project file format (OpenClap): (import { ClapWorkflow } from "@aitube/clap"))

  2. A .clap project (import { ClapProject } from "aitube/clap") can now embed an array of generic workflows (clap.workflows = [] // type ClapWorkflow[])

  3. A timeline segment (import { ClapSegment } from "aitube/clap") can now optionally reference a workflow id (segment.workflowId = "..."). Right now this will do nothing, but the idea is that the workflow should be called to generate the corresponding segment's output value. (I advise to use this with parcimony, to avoid using +100 different workflows within your project (you could, but I'm not sure it would be wise). Instead I encourage to use a default workflow for image, another default workflow for video, another for sound, music etc..)

  4. A workflow can have an array of parameters, and default values

  5. there is now a empty WorkflowEditor, to be worked on later. The idea is to add a UI to be able to visualize, edit and set the default parameters of workflows (again it's not about ComfyUI but any AI workflow engine like Transformers.js, MediaPipe, Visual Blocks, Glif etc)

sendn0des commented 3 months ago

love the idea of specific workflows. will avoid the comfy ecosystem compatibility and deprecated node issues.

glif could be useful here since the workflows are managed internally and maintained

for later, comfydeploy looks interesting for rapid iteration. it has a nice desktop workflow -> deploy -> frontend setup

jbilcke-hf commented 2 months ago

I have made some improvements to the code to improve ComfyUI support (a real ComfyUI server that you run locally or on your own remote server)

I believe we are close to having a basic (static) demo soon, hopefully this week

nha1012 commented 1 month ago

/boss $300

Himasnhu-AT commented 1 month ago

Is this issue still active?

devniel commented 1 month ago

Hi, I think so, there was already a progress done to target a local ComfyUI server.