ComfyUI_omost

ComfyUI implementation of Omost, and everything about regional prompt.

News

• [2024-06-10] Add OmostDenseDiffusion regional prompt backend support (The same as original Omost repo) https://github.com/huchenlei/ComfyUI_omost/pull/27
• [2024-06-09] Canvas editor added https://github.com/huchenlei/ComfyUI_omost/pull/28
• [2024-06-09] Add option to connect to external LLM service https://github.com/huchenlei/ComfyUI_omost/pull/25

TODOs

• Add a progress bar to the Chat node
• Implement gradient optimization regional prompt
• Implement multi-diffusion regional prompt

How to use

As you can see from the screenshot, there are 2 parts of omost:

• LLM Chat
• Region Condition

LLM Chat

LLM Chat allows user interact with LLM to obtain a JSON-like structure. There are 3 nodes in this pack to interact with the Omost LLM:

• Omost LLM Loader: Load a LLM
• Omost LLM Chat: Chat with LLM to obtain JSON layout prompt
• Omost Load Canvas Conditioning: Load the JSON layout prompt previously saved

Optionally you can use the show-anything node to display the json text and save it for later. The official LLM's method runs slow. Each chat takes about 3~5min on 4090. (But now we can use TGI to deploy accelerated inference. For details, refer Accelerating LLM.)

Examples:

• Simple LLM Chat:
• Multi-round LLM Chat:

Here is the a sample JSON output used in the examples:

Region condition

According to https://github.com/lllyasviel/Omost#regional-prompter, there are 6 ways to perform region guided diffusion.

Method 1: Multi-diffusion / mixture-of-diffusers

These method run UNet on different locations, and then merge the estimated epsilon or x0 using weights or masks for different regions.

TO be implemented

Method 2: Attention decomposition

lets say attention is like y=softmax(q@k)@v, then one can achieve attention decomposition like y=mask_A softmax(q@k_A)@v_A + mask_B softmax(q@k_B)@v_B where mask_A, k_A, v_A are masks, k, v for region A; mask_B, k_B, v_B are masks, k, v for region B. This method usually yields image quality a bit better than (1) and some people call it Attention Couple or Region Prompter Attention Mode. But this method has a consideration: the mask only makes regional attention numerically possible, but it does not force the UNet to really attend its activations in those regions. That is to say, the attention is indeed masked, but there is no promise that the attention softmax will really be activated in the masked area, and there is also no promise that the attention softmax will never be activated outside the masked area.

This is the built-in regional prompt method in ComfyUI. Use Omost Layout Cond (ComfyUI-Area) node for this method.

There are 2 overlap methods:

• Overlay: The layer on top completely overwrites layer below
• Average: The overlapped area is the average of all conditions

Method 3: Attention score manipulation

this is a more advanced method compared to (2). It directly manipulates the attention scores to make sure that the activations in mask each area are encouraged and those outside the masks are discouraged. The formulation is like y=softmax(modify(q@k))@v where modify() is a complicated non-linear function with many normalizations and tricks to change the score's distributions. This method goes beyond a simple masked attention to really make sure that those layers get wanted activations. A typical example is Dense Diffusion.

This is the method used by original Omost repo. To use this method:

• Install https://github.com/huchenlei/ComfyUI_densediffusion
• Include Omost Layout Cond (OmostDenseDiffusion) node to your workflow

Note: ComfyUI_densediffusion does not compose with IPAdapter.

Method 4: Gradient optimization

since the attention can tell us where each part is corresponding to what prompts, we can split prompts into segments and then get attention activations to each prompt segment. Then we compare those activations with external masks to compute a loss function, and back propagate the gradients. Those methods are usually very high quality but VRAM hungry and very slow. Typical methods are BoxDiff and Attend-and-Excite.

To be implemented

Method 5: Use external control models like gligen and InstanceDiffusion

Those methods give the highest benchmark performance on region following but will also introduce some style offset to the base model since they are trained parameters. Also, those methods need to convert prompts to vectors and usually do not support prompts of arbitary length (but one can use them together with other attention methods to achieve arbitrary length).

To be implemented

Method 6: Some more possible layer options like layerdiffuse and mulan

To be implemented

Optionally you can also pass the image generated from Omost canvas as initial latent as described in the original Omost repo:

Edit Region condition

You can use the built-in region editor on Omost Load Canvas Conditioning node to freely manipulate the LLM output.

Accelerating LLM

Now you can leverage TGI to deploy LLM services and achieve up to 6x faster inference speeds. If you need long-term support for your work, this method is highly recommended to save you a lot of time.

Preparation: You will need an additional 20GB of VRAM to deploy an 8B LLM (trading space for time).

First, you can easily start the service using Docker with the following steps:

port=8080
modelID=lllyasviel/omost-llama-3-8b
memoryRate=0.9 # Normal operation requires 20GB of VRAM, adjust the ratio according to the VRAM of the deployment machine
volume=$HOME/.cache/huggingface/hub # Model cache files

docker run --gpus all -p $port:80 \
    -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:2.0.4 \
    --model-id $modelID --max-total-tokens 9216 --cuda-memory-fraction $memoryRate

Once the service is successfully started, you will see a Connected log message.

(Note: If you get stuck while downloading the model, try using a network proxy.)

Then, test if the LLM service has successfully started.

curl 127.0.0.1:8080/generate \
    -X POST \
    -d '{"inputs":"What is Deep Omost?","parameters":{"max_new_tokens":20}}' \
    -H 'Content-Type: application/json'

Next, add an Omost LLM HTTP Server node and enter the service address of the LLM.

For more information about TGI, refer to the official documentation: https://huggingface.co/docs/text-generation-inference/quicktour

Using llama.cpp HTTP Server

Utilizing a quantized GGUF model can also lead to significant performance improvements.

1. Download the Model

   You can download the omost-llama-3-8b-Q8_0-GGUF model here. Alternatively, you can quantize other models yourself using the GGUF-my-repo.

2. Start the HTTP Server

   Clone and compile llama.cpp. Then, start the service with the following command:

   ./llama-server -m ../model/omost-llama-3-8b-q8_0.gguf -ngl 33 -c 8192 --port 8080

   Alternatively, you can directly use llama-cpp-python to start an HTTP Server.

   Reference: llama-cpp-python OpenAI-Compatible Web Server

   However, in my tests on the 3090Ti, using llama-server directly yields better performance. When generating an image, using llama-cpp-python takes approximately 50-60 seconds, whereas using llama-server takes around 30-40 seconds.

3. Add a Node

   Add an Omost LLM HTTP Server node and enter the service address of the LLM.