Separation of UI presentation and graph

[space-nuko]

Alot of visual programming tools I've used have a presentation mode separate from the graph mode, this is to reduce clutter and "wire forest" problems introduced by adding lots of complex nodes. Some examples:

Reaktor


Max/MSP

 

Would this paradigm work for comfyUI? It would be nice to have a "build your own webui" paradigm where you can hide the node graph completely and work only with the UI you've built, once you've perfected the layout of your nodes

This may increase the frontend's complexity depending on the limitations of pure JS and may be an unacceptable change, but I would like using comfyUI a lot more if it were somehow implemented

Related https://github.com/comfyanonymous/ComfyUI/issues/278#issuecomment-1485975122

[space-nuko]

Just noticed https://github.com/comfyanonymous/ComfyUI/issues/42. I would support separating the frontend/backend in an obvious way to support building this kind of alternative UI

[comfyanonymous]

Yes if there was a way to "compile" a node graph into an interface it would be a nice feature.

I would support separating the frontend/backend in an obvious way to support building this kind of alternative UI

The backend and frontend are already clearly separated.

Here's a simple example for how to use the api: https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

Here's an example of an alternative frontend: https://github.com/rvion/CushyStudio

[DivinoAG]

The way Blender handles that might be a good principle to follow. There it is possible to create a group of nodes that can be collapsed into a single node in the main node tree (which is also a good idea here, but another subject). But to allow passing parameters the inner nodes, there are very basic nodes called Group Input and Group Output, which only function is to create an interface for the external view of that group.

For example, on the node tree below you have some coordinate inputs added to the Group Input, and the Fac property is linked to the Group Output

And that makes it so when you view the group from the parent node tree, you see a nice interface showing all the parameters selected:

Now, on ComfyUI, you could have similar nodes that, when connected to some inputs, these are displayed in a sidepanel as fields one can edit values without having to find them in the node workflow. All this UI node needs is the ability to add, remove, rename, and reoder a list of fields, and connect them to certain inputs from which they will inherent all values and formatting from. They don't require any crazy UI, you can display them mostly one below the other, similar to the UI for Easy Diffusion:

[space-nuko]

I'm working on a PoC implementing this idea, code is here (working title) https://github.com/space-nuko/PlacidUI

[justanothernguyen]

IMO we just need portal nodes, i.e. connecting to this one node will also connect to the other node somewhere else. A "wireless" node, if you will. After which a "control panel" can be made using such nodes.

Right now with reroute nodes and a bit of diligent, lots of visual clutters can already be done away with

From this

to this

[space-nuko]

That's what Max/MSP's presentation mode does, it lets you save a separate version of your graph with the wires erased, and when you go back into edit mode it restores all the wires/node positions Either approach is fine I think, but the latter would require some serious hacking on litegraph to make it work. (Even with the former approach, litegraph's API is missing a few things that would make it easier like synchronizing node/widget state changes)

[Davemane42]

liteGraph does have support for subgraph but it is janky at best and multiple people already tried to implement it with mixed results.

Also, the "portal" node already exist as a custom extension (set/get node) but it keep breaking people workflows so 6ou might want to wait a bit.

A way to pin nodes/primitive to a collapsable side panel could be an easy way to display/change value on a big graph

[space-nuko]

I've been thinking also, litegraph is mostly canvas so it's not as good for accessibility as DOM elements. Plus with no way to anchor the node layout to the screen/prevent scrolling and zooming it would probably be tough to use as a standalone UI

[skyler14]

To add on the ideas here, it might make sense to make nodes and node groups all able to be activated/focused/etc. As ComfyUI gains prominence we'll see more tools trying to interface with SD beyond extensions and I think being able to then expand a graph node to a custom UI implementation may really have a lot of benefits.

[space-nuko]

Yeah I was thinking of being able to assign nodes "properties" like in a sidebar panel that affects things like grouping. So you could store the UI for each set of nodes in a separate tab, while they're all a part of the same graph. Or put them in an accordion.

Another thing, is being able to separate the individual widgets (sliders etc.) for each node panel and arrange them in a custom layout. I was thinking something like foobar2000's panel editing mode based on horizontal/vertical splitters, that translates to flexbox rows/columns. And defaulting to an alternative N-column layout for mobile users, both of which are saved to the workflow state. Since in theory webui's user interface is primarily based on an immutable layout of gr.Row/gr.Column, meaning fairly simple flexbox elements, that could be made editable with enough effort

[skyler14]

As a foobar2000 user I know what you mean.

I was thinking like this as far as how it could be an intuitive experience with nice extensibility

node groups: -have a default (logical) interface which is just a list of all the inputs and outputs of the components in them -you can define a (logical) interface that presents what inputs and outputs appear as edges on the node group -when you double click/press edit/want to enter edit mode you expand your node group and are able to adjust the internal components in a draft to save

nodes: -have a default (visual) interface which is just a list of all (the default to visible) options in them -you can define a (visual) interface that is basically made to treat the backend stuff making up that node like an api, it defaults to expanding to show a list of everything (like some kind of map of all the variables as some kind of input field + formatting), but just as easily the variables can be fed into a script to create a more elaborate UI (picture like someone had an viewer node for checking the results of an intermediate upsampling step and they wanted to add a image compare slider). -when you double click on an element either the default visual interface pops up or any custom interfaces you have loaded up/prepared/uploaded

[space-nuko]

Hmm, sounds interesting, I'm having some trouble visualizing what this would look like though. Could you provide some mockups if it's not too much trouble?

Your idea for a default interface sounds like the "mobile version" thing I had in mind, it's just a set of columns that can be collapsed into one single column on mobile or multiple on desktop depending on window size

[DivinoAG]

I think there are many different ways to approach this concept described by @skyler14.

The Node Groups UI could work exact as my previous example from Blender. You select a number of nodes and create a new group from them (Ctrl+G, or right clicking and selecting Create Node Group from a menu, for example). Every group automatically have Group Inputs and Group Outputs nodes. You plug any cable onto the empty connector on the Group Input/Output, and it adds a new parameter to that group. Double click that input/output name to rename them to whatever for clarity. When you close that group, only the parameters on those Input/Output nodes get displayed in the new group node. I'm posting the image here again for context.

As for the general node UI, the simplest way to achieve this I think would be to automatically display every parameter available to the user in a sidepanel, grouped by the node they belong to, and in the order those nodes are processed when you run the prompt. Then, give the user the option to hide the nodes/parameters they don't need to see for that workflow, so they end up with only the list of parameters they need to change in order to make their images.

I made a quick mockup of how I see that working, using my image upscaling workflow as an example:

Side panel showing all the editable parameters

Editing the visible parameters

Side panel showing only the selected parameters

Now, there is more that could done with this side panel, but I think this is an MVP that could get the ball rolling, and isn't terribly complicated to both develop, and to improve usability for users. I also didn't do anything too crazy in design here, mostly it's the same look we know and are used to. It could probably be improved, but that isn't truly necessary to make this work.

I also didn't address in this mockup how to handle Preview and Save Image nodes, which I think should probably replace the entire graph area there when viewing that side panel, with all images in a gallery style UI. I can make a mockup of that as well if that's something interesting to others.

[space-nuko]

@DivinoAG I've actually done this already with my alternative UI, except not in a sidebar but split into top and bottom. Nothing stopping it from being moved there in my mind. I made it like that so the graph can be hidden on a large screen and the UI can take up the whole space, since I expect with some workflows you'll never have to edit the graph once it's finalized

https://github.com/space-nuko/ComfyBox

That implementation puts the gallery in the UI itself but I could support pinned nodes or a specialized sidebar/overlay for displaying past outputs. Id like something with thumbnails and grouped by input workflow/generation/other too.

[0xdevalias]

Just stumbled onto this other issue about subgraphs while trying to find this one again, seems to be a lot of crossover potential in ideas and implementations being discussed between them:

• https://github.com/comfyanonymous/ComfyUI/issues/42
• https://github.com/comfyanonymous/ComfyUI/issues/497
• https://github.com/comfyanonymous/ComfyUI/issues/669
• https://github.com/comfyanonymous/ComfyUI/pull/724
• https://github.com/comfyanonymous/ComfyUI/pull/931
• https://github.com/comfyanonymous/ComfyUI/issues/1132
• https://github.com/comfyanonymous/ComfyUI/pull/1310
• https://github.com/comfyanonymous/ComfyUI/issues/1355
• https://github.com/comfyanonymous/ComfyUI/issues/1412
• https://github.com/comfyanonymous/ComfyUI/pull/1776
• https://github.com/ltdrdata/ComfyUI-Workflow-Component

  • This is a side project to experiment with using workflows as components.

---

Also wanted to include this idea/note I posted on the other issue here; RE: litegraph bugs/limitations and a potentially better alternative:

I think it will require several changes in litegraph to fix bugs.

These changes are in my fork of litegraph which isn't API compatible with upstream litegraph however as the pace of development is too significant

This is probably a little off topic for this issue, and I suspect that it would be a rather major / potentially breaking change to do so.. but I'm curious if there's a reason that litegraph is used instead of something like xyflow / reactflow / etc? I haven't deeply evaluated both projects yet, but at least in my initial cursory exploration, xyflow seems like it's capable of everything liteflow does, but in a more standard are modern developed/maintained way.

• https://github.com/xyflow/xyflow

  • React Flow | Svelte Flow - Powerful open source libraries for building node-based UIs with React (https://reactflow.dev) or Svelte (https://svelteflow.dev). Ready out-of-the-box and infinitely customizable.

  • https://www.xyflow.com/

    • Powerful open source libraries for building node-based UIs with React or Svelte. Ready out-of-the-box and infinitely customizable

  • https://reactflow.dev

    • Wire Your Ideas with React Flow A customizable React component for building node-based editors and interactive diagrams

  • https://svelteflow.dev

    • Wire Your Ideas with Svelte Flow A customizable Svelte component for building node-based editors and interactive diagrams by the creators of React Flow

Originally posted by @0xdevalias in https://github.com/comfyanonymous/ComfyUI/issues/669#issuecomment-1833179664

[ZapAndersson]

The "presentation mode" of Max/MSP is much much simpler, though. It's a mode of a whole graph, and when it is on, the only differences are

• only nodes flagged for "presentation mode" are displayed (the rest turn invisible)
• nodes move to an alternate "presentation mode" position

That's it! In theory, this should be pretty simple to implement, no?