search

comfyanonymous / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
53.21k stars 5.63k forks source link

feat: SetArea for models or LoRAs #806

Open jjangga0214 opened 1 year ago

jjangga0214 commented 1 year ago

LoRA tends to have side effects. For example, a fashion LoRA can affect the face or picture style, and so on.
To avoid this, there's an (A1111) extension hako-mikan/sd-webui-lora-block-weight, which limits the activation of layers selectively. However this also has side effects and requires a lot of experiments to find a suitable combination. Ideally, I think the best possible idea would be enabling SetArea for models and LoRAs, like conditioning. (Not block weight) So for instance, fashion LoRA can be only applied to body, not face. How do you think? Is this technically possible? Thanks :)

ltdrdata commented 1 year ago

Actually.. I'd implemented this once. https://github.com/comfyanonymous/ComfyUI/pull/303

However, it is expected that if the engine is overhauled for SDXL support, it will no longer be compatible, resulting in the postponement of the PR implementation. Now that the SDXL support work has been completed, it appears that maintenance work for PR will be possible, but it seems that I will have to take another look at the internal structure of the engine. lol.

jjangga0214 commented 1 year ago

@ltdrdata Thank you! That's also a really good node!

However, you misunderstood what I meant.

However this also has side effects and requires a lot of experiments to find a suitable combination. Ideally, I think the best possible idea would be enabling SetArea for models and LoRAs, like conditioning.

This means I imagine something like this :)

스크린샷 2023-06-27 오후 12 04 18
ltdrdata commented 1 year ago

Ah.. I just mentioned about lora block weights.

Maybe area wise lora application requires quite modification of sample.

Bocian-1 commented 1 year ago

Impact Pack allows you to apply lora/checkpoints to separate areas with no overlap with the TwoSamplersForMask nodes. It isn't perfect, but works

Using the TwoSamplersForMask node itself gives errors, but you can use the Iterative Upscale node with upscale_factor set to 1.001. 1.0 doesn't work. It will round the dimensions to multiples of 8 so unless you make some insane resolutions on the first pass it will keep the size as you set it.

You will also need to use the openpose controlnet to keep the character consistent and this requires some extra work to implement in a usable way.

It would be awfully convenient if someone happened to have already figured it out, wouldn't it? https://civitai.com/models/21100?modelVersionId=104147 image There are some other custom nodes used too, so make sure to check for them in About this version

ltdrdata commented 1 year ago

Using the TwoSamplersForMask node itself gives errors, but you can use the Iterative Upscale node with upscale_factor set to 1.001. 1.0 doesn't work.

What specific error are you encountering? Providing a description will be helpful in addressing the bug and assisting with the fix. It's possible that the type errors you encountered a few days ago were already fixed. Please give it another try on latest version and see if the issue persists.

Bocian-1 commented 1 year ago

What specific error are you encountering? Providing a description will be helpful in addressing the bug and assisting with the fix. It's possible that the type errors you encountered a few days ago were already fixed. Please give it another try on latest version and see if the issue persists.

I don't remember what it was, but it seems to be fixed now as I can't reproduce it. Thank you.

ltdrdata commented 1 year ago

Taking inspiration from this, I have added a node TwoAdvancedSamplersForMask to the Impact Pack.

https://www.youtube.com/watch?v=S1ksTNs5VX0