[Feature Request] Attention Mask for LORAs?

[MoonMoon82]

Is it possible to add a Node to mask the area where a specific LORA will be applied to? Matteos IPAdapter Node has got such a "attention mask". I found it very useful and it also would be greate to restrict the influence area of LORAs. (I know, it's already similar by using "SetLatentNoiseMask", but it would need to add sample processes each LORA - and it's just not the same)

Idk if it's even possible to implement such mask, but at least I want to ask for it.

[jakeute]

masks are useful when there are multiple loras

[MoonMoon82]

@jakeute Usually the output of a sampler is not very predictable, unless you're using controlnet modules to predefine the shape of the output. So in this case, masked loras in conjunction with controlnet modules would also be usefull.

[suede299]

The IPAdapter's ATTENTION MASK is the reference portion of the target. For lora to work like this, it needs to be targeted to the generated image, but the images are all still in the process of being generated, so how do you know where a certain lora should be applied? To achieve such an effect, I'm afraid that 2 samplers are needed, applying ConditioningSetMask, SetLatentNoiseMask to the 2nd sampler.

[MoonMoon82]

@suede299 yes, that's exactly how IPAdapters "attention mask" works. I'm totally aware of that it already can be achieved with 2 samplers in confunction with nodes like SetLatentNoiseMask. But the disadvantage of course is the increase of complexity and decrease of performance of your workflow. Using this method IPAdapters "attention mask" would also be unnecessary.

You can be pretty sure where a certain lora should be applied to, as soon you're using controlnet like canny, openposen or segmentation controlnet for example.

I really don't have an idea if this is even possible. But I know that this would be very helpful!

[manzonif]

here an implamentation for sd-webui

[MoonMoon82]

@manzonif That's awesome! That means it is technically possible to implement such a feature?

Is there anyone out there who is capable of creating such a node out of this code? To mask the target area of a lora would be very very helpful!

Unfortunately I don't even have an idea where to start and I can even barely read python :(

[manzonif]

technically yes. I'm new to python, but I want to spend a few hours trying to implement it. I will hardly succeed :-(

[MoonMoon82]

@manzonif That would be great! If I can help you in any kind of way please let me know! (e.g. testing a node)

[MoonMoon82]

It seems there is an addon for A1111 to mask the output of LORAs: https://github.com/lifeisboringsoprogramming/sd-webui-lora-masks

I would really really appreciate if someone could make a lora-attention-masking node for comfyui!

[MoonMoon82]

I still have some hope this attention mask node could be developed..

[AlexD81]

+1

[MoonMoon82]

My hope is fading :-(

[MoonMoon82]

Hi @shawnington , how does this commit (https://github.com/comfyanonymous/ComfyUI/commit/0fecfd2b1a2794b77277c7e256c84de54a63d860) solve the missing attention mask for LORAs ?

[shawnington]

Hi @shawnington , how does this commit (0fecfd2) solve the missing attention mask for LORAs ?

That is the issue number 2445 is for the PIL library, it was not intended to be linked to this feature request.

[MoonMoon82]

@AlexD81 , @manzonif , @jakeute, @suede299

I found these two custom nodes in AnimageDiffEvolved which seem to do what I was searching for: ADE_RegisterLoraHook

ADE_ConditioningSetMaskAndCombine

The first node loads the Lora but the lora is only activated for a specific masked area in the second node. At the second node the Cond_ADD can be an empty/ZeroOut condition or a lora specific "trigger word". All my tests confirm that this works the way I requested.

Could anyone also confirm?

Maybe this issue can be closed then.

[Phando]

Do the LoraHooks only work with animated diff models? I am trying to add ADE_RegisterLoraHook and ADE_ConditioningSetMaskAndCombine with an SDXL workflow. Is it possible to use a standard sampler or do I need to set up a workflow with an AnimatedDiff sampler?

My goal is to generate an image with two characters, each coming from a different lora in the same generation. Regional conditioning is working, but the loras seem to be unmaskable.

[MoonMoon82]

@Phando Yes, it works with usual Loras as well. Here's a basic workflow that can show you how it works in general: multiple masked Loras.json

I had an issue in creating separate face masks, so I did it manually (upper right corner).

In this example the Loras have to same trigger words. So if your Loras have different trigger words, you should set up two different ADD_conditions. I'm not quite sure if I handle it right, but to me it seems it works just fine.

[Phando]

Thank you for the sample workflow! It looks like my Comfy was out of date, I kept getting tuple errors when the sampler was running. Your workflow helped me figure it out. Now it is time to have some fun with regional loras!

[Phando]

It worked! By adding regional conditioning in conjunction with the LoRA hooks, its looking pretty sweet. Thank you. [Uploading workflow (2).png…]()