webui-forge Support

[StylusEcho]

Hi, lllyasviel released a version of Automatic1111 called Forge. Currently, loractl doesn't work with it. I get the following:

*** Error loading script: loractl.py
    Traceback (most recent call last):
      File "C:\Machine Learning\Automatic1111 Forge\modules\scripts.py", line 541, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "C:\Machine Learning\Automatic1111 Forge\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "C:\Machine Learning\Automatic1111 Forge\extensions\sd-webui-loractl\scripts\loractl.py", line 5, in <module>
        from loractl.lib import utils, plot, lora_ctl_network, network_patch
      File "C:\Machine Learning\Automatic1111 Forge\extensions\sd-webui-loractl\loractl\lib\plot.py", line 6, in <module>
        from loractl.lib.lora_ctl_network import networks
      File "C:\Machine Learning\Automatic1111 Forge\extensions\sd-webui-loractl\loractl\lib\lora_ctl_network.py", line 10, in <module>
        import network, networks, network_lora, extra_networks_lora
    ModuleNotFoundError: No module named 'network_lora'

---

However, lllyasviel claims Forge makes extension development easier and more efficient. Could you please check out Forge and see what you can do? Thank you.

[altoiddealer]

So I followed the traceback, and found that the script was looking for /extensions-builtin/Lora/network_lora.py

I don't see it in Forge... so I go ahead and copy/paste it from A1111. I restart Forge and next it is missing /extensions-builtin/Lora/lyco_helpers.py so again, I go ahead and copy/paste it from A1111.

EDIT Nevermind, there are no errors on launch but the extension is not actually working.

[311-code]

I would also love to see this working in forge, and also working with adetailer because adetailer doesn't seem to recognize the loractl syntax.

[cheald]

Unfortunately, loractl is very intimately tied to the innards of how A1111 implements LoRAs. The concept could certainly be applied in other tools like Forge or ComfyUI, but the actual implementation is likely to be wildly different depending on the innards of their lora apply code. It would better for a loractl-like extension for each project to be its own distinct project, just because there would be very little actual shared code between them. In many ways, this is less of a pure "extension" and more of a ugly monkeypatch on top of A1111's specific codebase.

A1111 makes this fairly easy because it already has support for variant lora weights (as do most), and it runs the lora network weight setup process with each step -- the trick is that if the lora being changed haven't changed since the previous step, it doesn't have to do the expensive work of reverting to the base model weights and recomputing the lora weights to apply.

loractl hijacks that process and invalidates A1111's cache, and "lies" about what the weight should be, by dynamically computing it for each step. This just lets the native A1111 lora application code do the rest.

A1111 also supports a generalized "extra networks syntax", which is what allows this extension to cleanly replace the existing syntax. Rather than trying to do everything itself, it just extends what A1111 already provides.

The extension is just mutating A1111's internal mechanisms, and heavily relies on the specifics of their implementation. This is why it's not particularly viable to just tweak it for other platforms - the extension would have to be fundamentally rewritten for each of them.

It's been a while since I've looked at this stuff, but glancing over Forge's code, it looks like it's using something closer to the pre-1.5 A1111 implementation of loras - which this extension will specifically not work with, since it depends on the 1.5 rewrite of lora handling. You might try the composable lora extension, which this one takes its inspiration from, and which worked with the pre-1.5 A1111 webui.

That said, I highly welcome and encourage forks or derivations of this project for platforms like Forge or ComfyUI. I haven't done very much at all with Stable Diffusion in the last few months, and am somewhat out of the loop regarding the developments in the space over that period, and I don't have a lot of time right now to put towards new similar projects, but I certainly hope that someone else will take the idea and run with it.

[BurnZeZ]

@cheald What changed in the rewrite and did forge truly stick with the older version? What exactly are the implementation-specific details?

[cheald]

I wrote up more detail here: https://github.com/lllyasviel/stable-diffusion-webui-forge/issues/68#issuecomment-1945491969

tl;dr Forge appears to set up lora weights once at the start of rendering (which is intelligent, since it's slow and doesn't usually change mid-render), but since it doesn't have hooks for recomputing lora weights per step, loractl's approach can't work.

loractl works by invaliding A1111's lora cache, which causes the A1111 lora code to reapply new lora weights on each step. This involves restoring the base weights and then recomputing the new weights with the lora weights times some weight multiplier. This is why loractl slows down rendering as much as it does - this isn't a quick process, though I'm sure that the Forge authors can probably find ways to make it a bit more optimized.

[rterreereeeee]

PLS