Open jbilcke-hf opened 4 months ago
/boss $250
Total Bounty: $1050
Flexibility to pull from openart.ai would present some wonk. There's too many workflows and many are meh - but it could lead to an issues with frontend design later attempting to accommodate.
I'll bring over some workflows I've considered or have that would play well and can design toolkit around them.
/boss $500
I'm making some changes to the codebase to prepare and facilitate work for this ticket.
These changes are not specific to ComfyUI, so not in the scope of this ticket, and not meant to claim the bounty:
I've added the concept of workflow to Clapper and the project file format (OpenClap):
(import { ClapWorkflow } from "@aitube/clap")
)
A .clap project (import { ClapProject } from "aitube/clap"
) can now embed an array of generic workflows (clap.workflows = [] // type ClapWorkflow[]
)
A timeline segment (import { ClapSegment } from "aitube/clap"
) can now optionally reference a workflow id (segment.workflowId = "..."
). Right now this will do nothing, but the idea is that the workflow should be called to generate the corresponding segment's output value. (I advise to use this with parcimony, to avoid using +100 different workflows within your project (you could, but I'm not sure it would be wise). Instead I encourage to use a default workflow for image, another default workflow for video, another for sound, music etc..)
A workflow can have an array of parameters, and default values
there is now a empty WorkflowEditor, to be worked on later. The idea is to add a UI to be able to visualize, edit and set the default parameters of workflows (again it's not about ComfyUI but any AI workflow engine like Transformers.js, MediaPipe, Visual Blocks, Glif etc)
love the idea of specific workflows. will avoid the comfy ecosystem compatibility and deprecated node issues.
glif could be useful here since the workflows are managed internally and maintained
for later, comfydeploy looks interesting for rapid iteration. it has a nice desktop workflow -> deploy -> frontend setup
I have made some improvements to the code to improve ComfyUI support (a real ComfyUI server that you run locally or on your own remote server)
I believe we are close to having a basic (static) demo soon, hopefully this week
/boss $300
Is this issue still active?
Hi, I think so, there was already a progress done to target a local ComfyUI server.
Context
ComfyUI is all the rage right now, and I would like to use it as one of the supported backends for Clapper.
Currently, Clapper only has ultra-basic support for ComfyUI cloud providers, such as Comfy.icu (see this screenshot, the ugly text field at the very bottom):
Goal
At minimum, it should be able to use an existing integration running locally or in the cloud
But what I have in mind is more something like this:
Limitations / things to checks
I'm not a ComfyUI developer (or user), so it's gonna be a fun ticket!
How to implement
For now I've only added some basic code that doesn't really work (search for "comfy" and "ComfyIcu" - it's related to comfy.icu - in the codebase)
Acceptance criteria
Being able to run any ComfyUI workflow, for :
image
,video
, and why no alsosound
,voice
andmusic