TinyTerra / ComfyUI_tinyterraNodes

A selection of nodes for Stable Diffusion ComfyUI
GNU General Public License v3.0
421 stars 45 forks source link

Many ideas: Autocomplete "LoRA Tag", Autocomplete "preset text" #56

Closed mijuku233 closed 7 months ago

mijuku233 commented 1 year ago

This node is awesome and I want to help make it even better, but I'm not a programmer so I can only offer some ideas. Below are my ideas, which seem a bit complicated because I thought about them for several days.

automatic completion:

Autocomplete "LoRA Tag"

- After entering `@`, a pop-up window will be displayed to list the `LoRA list`. After selecting a `LoRA`, the `LoRA Tag` will be automatically filled in based on the format ``

Autocomplete "preset text"

- After entering `#`, the `preset text list` will be listed. After selecting a `preset text`, fill in the contents of `preset text.txt` into the current text box - Adds `Save as preset text` to the node right-click context menu > Save the contents of the text box as a `txt` file and add it to the `preset text list`

PipeLoder:

New feature: Load LoRA from text

- Read `LoRA tag(s)` from text and load it into checkpoint model. > Reference [comfyui_lora_tag_loader](https://github.com/badjeff/comfyui_lora_tag_loader) In fact, there are bugs in `Lora load` and `lora stack` of `PipeLoder`, but I think instead of fixing them, it is better to change to a more efficient LoRA loading method. - Add a widget:`enable`/`disable`LoRA, default value`enable`

New input: Append Positive and Append Negative

- Append entered text to `Positive`/`Negative`

New output: prompt word text

- Output the `prompt word` and exclude `LoRA Tag` and `Embedding` > ![Screenshot 2023-10-12 221114](https://github.com/TinyTerra/ComfyUI_tinyterraNodes/assets/89350747/f8d7b5a7-7343-4d36-aa85-8357fad62224)

PipeKSampler:

New input: image

- Supports encoding and sampling of input `image` > 1.`PipeEdit` and `PipeIN` can input `image`, but have no effect or report an error, perhaps because `PipeKSampler` does not support the input of `image`. > 2.`upscale by model` requires it - Add new component: sampling object "`Latent`/`image`", default value `image` > If the selected object has no input, process another object This widget allows people to quickly switch between `text2image`/`image2image`

Upscale improvements

- New component: hiresfixScale "`Disable`/`Direct amplification`/`Post-sampling amplification`", the default value is `Disable` > 1.Disable: Do not perform the `upscale` operation and hide related components > 2. Direct amplification: first perform `upscale` processing, then `sample`, suitable for `image2image` > 3. Amplify after sampling: first `sample` the latent/image, then perform `upscale` processing, and then `sample`, suitable for `text2image` - New widget: upscale model "`none`/`upscale model name...`", default value `none` > After selecting any `upscale model`, enable upscale by model. This upscaling method is usually better than upscaling latent.

New option for ‘image output’

- Disable `vae decode` > Sometimes `vae decode` is not needed, so add an option to disable `vae decode`. Another thing to note is that when `PipeKSampler` disables `vae decode` and does not connect the `save`/`preview` nodes, `PipeKSampler` should not run. This is a bug in `KSampler (Efficient)` .

other:

New node: pipe=>XLpipe

- Combine two `pipes` into `pipeSDXL` > Can replace `pipeLoaderSDXL`, `pipeLoaderSDXL` complicates simple things

Improve the LoRA part of xyplot

- Can be simplified to a single `lora option`, no need for `lora1` `lora2`..., nor need to distinguish between `name`/`model weight`/`clip weight` - You can use any combination of `LoRA` for comparison, such as: > `` ``; `` ``; ``; x1: For both `LoRA`, `model weight` is 1 x2: The same two `LoRA` as x1, `model weight` 0.4, `clip weight` 0.7 X3: `lora name` is different and the quantity is also different

Combined nodes: PipeBus

- Combined by `pipeedit` and `pipeout` > The two can be combined together, making it more convenient to use

Some components that can be deleted

- The `seed component` in `pipeLoader` `pipeEdit` `pipeIN` `pipeOUT`, I think it is enough to only keep the `seed output` of `pipeKSampler` - The `LoRA component` in `pipeKSampler`, I just feel that there is no need to add LoRA in the sampler. After changing to a more efficient LoRA loading method, it can also be deleted.

After completion, the relevant nodes will almost look like this:

Node preview

![Screenshot 2023-10-12 223419](https://github.com/TinyTerra/ComfyUI_tinyterraNodes/assets/89350747/05afb114-d252-4fd9-b8f4-6af78ffe42c8)

TinyTerra commented 1 year ago

Always appreciate ideas to make it better :)

Haven't had too much time to work on the node pack lately but I'll add these to my list.

The autocomplete ideas are great, as well as the lora loading from text. - Just out of curiosity there, in what way are the current lora loader methods bugged? I think there's some model caching issues in the latest push that I need to figure out still.. might be related to that.

Not too sure where appending text will come in handy - but it definitely wouldn't hurt to add it. Text output sounds good too.

the pipe_line should already support image input so I'll need to do some debugging there if the ksampler doesn't errors out. But the idea of incorporating img2img directly into the ksampler sounds good too.

Definitely agree with pipeLoaderSDXL being overcomplicated - but there are certain parameters that are required that aren't present in the default pipeLoader. That being said you can still already do SDXL sampling with the standard pipeloader without using SDXL pipe at all, so its a bit of a preference thing. I'm open to fleshing out how to improve the SDXL side of things - Admittedly it was hastily put together, without too much SDXL specific experience.

I'm a little bit hesitant to outright delete or remove things, just for the sake of backwards compatibility with old workflows.

Adding Loras in the KSampler can be handy with things like add_detail, where you only want it to effect the upscale - doesnt make a lot of sense for lora that require keywords, but there are still some scenarios where its handy i think.

TinyTerra commented 1 year ago

TODO:

mijuku233 commented 1 year ago

Current lora loader bugs:

  1. The LoRA effect loaded by pipeLoader is incorrect:
  1. lora stack does not take effect:

lora stack

mijuku233 commented 1 year ago

The use of appending text:

For example, appending the prompt words interrogated by wd14 tagger to Positive

mijuku233 commented 1 year ago

If you want to retain the lora load in pipeKSampler, you can modify it to load lora from text. The advantage of this is that you can load any number of lora, just like Detailer (SEGS) in impact

Detailer (SEGS) 2023-10-14 154500

TinyTerra commented 1 year ago
  1. The LoRA effect loaded by pipeLoader is incorrect:
  • normal lora
  • pipeLoader lora

the difference you're seeing there is because of the lora clip. the pipeLoader lora applies the lora first and then creates the conditioning with the model+lora clip - so its not necessarily wrong its just being processed in a different order. That is also part of the intent of the loraStack input - so they can be loaded pre conditioning. but just having a text based input for loras would somewhat remove the need for the loraStack..

  1. lora stack does not take effect:

Just pushed a fix for that - it was broken, it was returning incorrectly when there wasn't a pipe_line connected. Should be all good now.

TinyTerra commented 7 months ago

V2 has added the following requests

Add Lora Autocomplete pop-up Update pipeLoader with text inputs and outputs Add hiresfix upscale method to pipeKSampler Add Disable option to image output of pipeKSampler Update lora logic in xyplot (xyplot has been deprecated for advanced xyPlot)