Use LORA's or Hypernet's

[Syncriix]

First of, amazing extention, loads of fun playing with it!

I came to wonder though, if it would be possible to use LORA's with it, as for example:

[<lora:someLora_v2:.6>:somekeyword:0,10]

However it won't affect generation, the lora remains enabled throughout.

I'm wondering whether it is the extra :, since it works with Textual Inversions or if LORA's simply can't be turned on/off.

I have tried to use it as variable to get rid of the :, but then again, that could be parsed before it reaches the fusion stage.

Thanks, cheers and good fortune to every reader <3

[ljleb]

Hi, thanks for the request!

The problem turns out to be that loras are loaded and stripped from the prompt before the extension has any chance of seeing them. See the related discussion: #45

At the moment, your example above is equivalent to:

<lora:someLora_v2:.6> [ : somekeyword : 0, 10]

as well as

[ : somekeyword : 0, 10] <lora:someLora_v2:.6>

In other words, the location of loras in a prompt does not affect the result.

Something that may be possible to be done however is to interpolate the weight of a lora. For example:

<lora:someLora_v2 : .6, 0>

I am not sure how to do this yet or if it is actually possible, I will look into it. It may be possible to convert the prompt interpolation syntax into this linear interpolation syntax, achieving a similar effect to what you are looking for.

[ljleb]

@Syncriix @caphalorthrow I just stumbled on this: https://github.com/a2569875/stable-diffusion-webui-composable-lora

Have a look at the readme, it seems they allow to interpolate lora weights through sampling steps. Their syntax is quite comprehensive, although it may be a bit hard to get the hang of it IMO. Maybe a good place to start for you guys.

[asagi4]

I managed to hack the composable lora extension and prompt fusion to sort of work together; it's really awful code though.

Basically I patched composable lora to check if prompt fusion exists and then uses importlib magic to get references to _parse_tensor_builders and _get_flattened prompts so that composable lora can see the "expanded" prompts that prompt fusion creates. This allows me to use [<lyco:xyz:1>:0.5] instead of being forced to specify steps "manually"

I feel like this should be a lot easier if prompt fusion did its prompt expansion bit earlier, but I guess that might be difficult to implement given that A1111's internal architecture is very messy, to put it lightly.

[ljleb]

Nice work! One way to implement this directly in prompt fusion seems to be to hijack the lora implementation ourselves and use the parsed prompts to infer the weight to apply to the lora file at each step.

I feel like this should be a lot easier if prompt fusion did its prompt expansion bit earlier

Prompt fusion will generate an exponential number of prompts depending on how many interpolations there are, which could be confusing for some scripts as usually len(p.all_prompts) == p.batch_size * p.n_iter. I think there needs to be a secondary hijack that deals with the lora parsing in isolation. Feel free to open a PR if you figure it out, I'll be glad to review!

[pmcculler]

If I understand the discussion, the extension https://github.com/cheald/sd-webui-loractl does this dynamic weighting, it's very useful and I use it all the time.

[ljleb]

I'll close this issue as the feature is available somewhere else. Please open another issue if you find a compatibility problem.