[Issue]: LoRa not working

[TomerGadol]

Issue Description

After loading LoRas from the 'extra networks' menu I receive an error: RuntimeError: output with shape [128, 320] doesn't match the broadcast shape [128, 320, 128, 320] The error occurs with different LoRa files, all in .SafeTesnor format, downloaded from civitai, for both 2.1 and 1.5 models.

Here's an image for more details:

Platform Description

Win10 on commit f256fb8b6ace49f201ea351edf9cf835dbb0fd62

[vladmandic]

I've just tried and do not see any errors. And you've not provided much of a platform details - at the very least, what is the version of the app you're using? Did you try to upgrade?

[razvan-nicolae]

looks like lora error when using a 1.5 lora on 2.1 model or viceversa maybe

[TomerGadol]

Not sure what happened but I just performed an update via git pull (b28b7c7a98a5e6cf88e9ecfb05ff844f72c14ce4), using the base 1.5 model, vae-ft-mse-840000-ema along with a 1.5 LoRa. Now it doesn't return an error but it seems the LoRa didn't affect the generation at all

[vladmandic]

when you generate with prompt that has lora, do you see lora being loaded in console log? (original a1111 does not log it, but this repo logs when lora gets loaded).

[TomerGadol]

Yes it is being loaded:

[vladmandic]

so what exactly is not working?

[TomerGadol]

The generation lacks any resemblance to the lora. For example, I'm using this lora from civitai and using similar parameters to those used in the example but getting results as if I didn't use the lora at all.

[vladmandic]

i don't see anything wrong. you may increase number of steps as lora's take a bit longer to take over the style. and you may need to change the base model you're applying it on, some loras are sensitive to that.

but having said, that, simple prompt "photo of a woman " with steps at 40 and using default sd1.5 model produced this - clearly not a default render as without a lora.

[univeous]

Try this one, it didn't work. It's a LyCORIS, but I believe this repo supports lyCORIS according to the readme. The base model is Anything 3.0.

This is a working sample, Lora is here and there's no error in the console output.

Log:

Loading weights: H:\AI\automatic\models\Lora\loconOfHoshinoAiOshi_v10.safetensors ------------------ 0.0/44.0 MB -:--:--
activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x0000025735EC5420>]:
RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ H:\AI\automatic\modules\extra_networks.py:75 in activate                                                             │
│                                                                                                                      │
│    74 │   │   try:                                                                                                   │
│ >  75 │   │   │   extra_network.activate(p, extra_network_args)                                                      │
│    76 │   │   except Exception as e:                                                                                 │
│                                                                                                                      │
│ H:\AI\automatic\extensions-builtin\Lora\extra_networks_lora.py:23 in activate                                        │
│                                                                                                                      │
│   22 │   │                                                                                                           │
│ > 23 │   │   lora.load_loras(names, multipliers)                                                                     │
│   24                                                                                                                 │
│                                                                                                                      │
│ H:\AI\automatic\extensions-builtin\Lora\lora.py:214 in load_loras                                                    │
│                                                                                                                      │
│   213 │   │   │   if lora is None or os.path.getmtime(lora_on_disk.filename) > lora.mtime:                           │
│ > 214 │   │   │   │   lora = load_lora(name, lora_on_disk.filename)                                                  │
│   215                                                                                                                │
│                                                                                                                      │
│ H:\AI\automatic\extensions-builtin\Lora\lora.py:176 in load_lora                                                     │
│                                                                                                                      │
│   175 │   │   with torch.no_grad():                                                                                  │
│ > 176 │   │   │   module.weight.copy_(weight)                                                                        │
│   177                                                                                                                │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: output with shape [4, 4, 1, 1] doesn't match the broadcast shape [4, 4, 3, 3]
Data shape for DDIM sampling is (1, 4, 96, 64), eta 0.0
Running DDIM Sampling with 50 timesteps
DDIM Sampler: 100%|█████████████████████████████████████████████████████████████████████| 50/50 [00:06<00:00,  7.25it/s]

Version:

updated: 2023-04-14
hash: bf949a53
url: https://github.com/vladmandic/automatic.git/tree/master

[TomerGadol]

Just to be clear, do you use "<>" to encapsulate the lora:lora_name:1 or is with it?

[exodia52]

Seems like this happened when using SD 2.1 Model with the Lora. If used with Sd 1.5 Model there's nothing wrong with the Lora.

[TomerGadol]

Well I've tried again using this prompt photo of a woman lora:LuisapHotlineStyle_v2:1 and got these results that look nowhere near yours. Using base 1.5 model and VAE. DDIM 40 steps.

[vladmandic]

well, there is something definitely in conflict when applying lora on your system:

activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x0000025735EC5420>]:
RuntimeError

do you have any additional lora-related extensions installed that may be causing conflict?
can you post extensions enabled line from startup log?

regarding lycoris/locon, i found one issue and i think i fixed it, new version is posted.

btw, to verify, example of two loras also works:

masterpiece, best quality, 1girl, solo, , smile, school uniform, computer, cyberpunk, cityscape, skyscraper

[TomerGadol]

The last post is from another user- @univeous, but here is my startup log:

[univeous]

It's still not working on ed8819b8. Here is the full log, the only extension I installed is a1111-sd-webui-tagcomplete.

❯ ./webui.bat --no-half-vae --upgrade
venv "H:\AI\automatic\venv\Scripts\Python.exe"
*unrelated* INFO     Python 3.10.7 on Windows
*unrelated* INFO     Running setup
*unrelated* INFO     Version: ed8819b8 *unrelated*
*unrelated* INFO     Updated to version: ed8819b8 *unrelated*
*unrelated* INFO     Updating Wiki
*unrelated* INFO     Installing requirements
*unrelated* INFO     Installing packages
*unrelated* INFO     Installing repositories
*unrelated* INFO     Installing submodules
*unrelated* INFO     Updating submodules
*unrelated* INFO     Extensions disabled: []
*unrelated* INFO     Extensions built-in: ['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora',
*unrelated*          'model-keyword', 'prompt-bracket-checker', 'ScuNET', 'sd-dynamic-thresholding',
*unrelated*          'sd-extension-aesthetic-scorer', 'sd-extension-steps-animation', 'sd-extension-system-info',
*unrelated*          'sd-webui-controlnet', 'sd-webui-model-converter', 'seed_travel',
*unrelated*          'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'SwinIR']
*unrelated* INFO     Extensions enabled: ['a1111-sd-webui-tagcomplete']
*unrelated* INFO     Setting environment tuning
*unrelated* INFO     Torch 2.0.0+cu118
*unrelated* INFO     Torch backend: nVidia CUDA 11.8 cuDNN 8700
*unrelated* INFO     Torch detected GPU: NVIDIA GeForce RTX 3070 Ti VRAM 8192 Arch (8, 6) Cores 48
*unrelated* INFO     Server arguments: ['--no-half-vae', '--upgrade']
*unrelated* INFO     Available models: H:\AI\stable-diffusion-webui\models\Stable-diffusion 32
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Loading weights: H:\AI\stable-diffusion-webui\models\Stable-diffusion\Anything-V3.0\Anything-V3.0.ckpt ---- 0.0… -:--:…
                                                                                                            GB
Creating model from config: H:\AI\automatic\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading weights: H:\AI\stable-diffusion-webui\models\Stable-diffusion\Anything-V3.0\Anything-V3.0.vae.pt ---- 0.0… -:-…
                                                                                                              MB
Applying scaled dot product cross attention optimization
Textual inversion embeddings loaded(67): *unrelated*
Model loaded in 10.6s (load=5.7s create=0.5s apply=0.7s vae=1.5s move=0.6s embeddings=1.5s)
00:24:55-908684 INFO     Startup time: 25.3s (torch=4.3s libraries=2.1s models=0.1s codeformer=0.2s scripts=4.1s
                         ui=3.3s gradio=0.2s checkpoint=11.0s)
Loading weights: H:\AI\automatic\models\Lora\loconOfHoshinoAiOshi_v10.safetensors ----------------- 0.0/44.0 MB -:--:--
Error activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at
0x0000017B5E51DA20>]: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ H:\AI\automatic\modules\extra_networks.py:75 in activate                                                            │
│                                                                                                                     │
│    74 │   │   try:                                                                                                  │
│ >  75 │   │   │   extra_network.activate(p, extra_network_args)                                                     │
│    76 │   │   except Exception as e:                                                                                │
│                                                                                                                     │
│ H:\AI\automatic\extensions-builtin\Lora\extra_networks_lora.py:23 in activate                                       │
│                                                                                                                     │
│   22 │   │                                                                                                          │
│ > 23 │   │   lora.load_loras(names, multipliers)                                                                    │
│   24                                                                                                                │
│                                                                                                                     │
│ H:\AI\automatic\extensions-builtin\Lora\lora.py:214 in load_loras                                                   │
│                                                                                                                     │
│   213 │   │   │   if lora is None or os.path.getmtime(lora_on_disk.filename) > lora.mtime:                          │
│ > 214 │   │   │   │   lora = load_lora(name, lora_on_disk.filename)                                                 │
│   215                                                                                                               │
│                                                                                                                     │
│ H:\AI\automatic\extensions-builtin\Lora\lora.py:176 in load_lora                                                    │
│                                                                                                                     │
│   175 │   │   with torch.no_grad():                                                                                 │
│ > 176 │   │   │   module.weight.copy_(weight)                                                                       │
│   177                                                                                                               │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: output with shape [4, 4, 1, 1] doesn't match the broadcast shape [4, 4, 3, 3]
Progress 3.5it/s ---------------------------------------- 100% 0:00:00 0:00:02

[vladmandic]

lora is clearly failing to load correctly, but everything seems normal before that - no weird extensions, no conflicts, torch and gpu are ok, etc...

only strange thing is that you're using --no-half-vae? any reason for that? it really should not be necessary on rtx3000 series.

but...loconOfHoshinoAiOshi_v10 is LOCON/LYCORIS, you should not load it as LORA.
it should be in models/LyCORIS path and prompt should be <lyco:loconOfHoshinoAiOshi_v10:1>.
and then console output should include locon load lyco method.

[univeous]

You're right, I'm sorry for that. After doing that the error is gone and everything works fine. Load a Locon as Lora works fine in the vanilla A1111 WebUI, and it was the standard behavior when Locon just came out, so I think it will be fine, but seems something was updated.

for the --no-half-vae, sometimes I will get a black image without it.

[vladmandic]

It used to be same loader as locon patched on top of lora, but as both were getting developed, it's started causing some incompatibilities so locon author decided to separate it (and I agree with thar approach)

[Shadowghost]

Since we can set the LoRA directory, wouldn't it make sense to add an UI option to set the LyCORIS directory too?

[exodia52]

Since we can set the LoRA directory, wouldn't it make sense to add an UI option to set the LyCORIS directory too?

It's already available. See the Setting > System Paths

[Shadowghost]

Huh, seems like I got a cached version fo the page last time I checked, Thanks for the heads-up!

[vladmandic]

seems issue is solved, so closing for now. if there are still any problems, let me know.

[TomerGadol]

Hey, glad that @univeous managed to solve the problem but the issue is still ongoing on my end.

[TomerGadol]

Well I just used git pull and now I get a different error altogether and can't even start the UI: HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/spaces/Default)

[vladmandic]

@TomerGadol ah, ok...

lets summarize:

• no errors during startup?
• no errors during runtime?
• lora loads correctly (noted in console output)?
• lora does not seem to do anything

and lets pick single model and lora to focus on so we don't have differences:

• model: v1-5-pruned-emaonly
• lora: https://civitai.com/models/7088/rihanna-lora
• prompt: photo of girl <lora:rihannaLora_v10>
• seed: 4082051732
• sampler: UniPC
• steps: 20

my output is:

[TomerGadol]

Well, now I can't even start the webui beacuse of the error I'm getting. Should I move this to another issue and return to this one after that is resolved?

[vladmandic]

sure.

[TomerGadol]

So back on track:

• no errors during startup
• no errors during runtime
• lora loads correctly (noted in console output)
• lora does not seem to do anything

Same specs:

• model: v1-5-pruned-emaonly
• lora: https://civitai.com/models/7088/rihanna-lora
• prompt: photo of girl <lora:rihannaLora_v10>
• seed: 4082051732
• sampler: UniPC
• steps: 20

My output is pretty much the same:

When I add the LuisaP lora as photo of girl <lora:rihannaLora_v10:1> <lora:LuisapHotlineStyle_v2:1> I get this output which isn't affected by said lora even tho it is being loaded correctly in the console:

So I guess the issue is with that specific lora, but it is listed as 1.5 and you seem to make it work. I'll try testing with other loras as well.

[vladmandic]

seems like there is no core issue, so i'll close this issue. but i'm curious what you find, so feel free to keep updating here.

[TomerGadol]

The weird thing is that other loras by the same creator are working fine. I'll try redownloading the problematic lora and see if it helps.

[vladmandic]

author probably used a different method to create lora, i'd be very curious to know how it was created.

[TomerGadol]

I've posted on Civitai and hope she'll respond soon. But it still doesn't explain how you manged to get these results and I can't seem to replicate that. I've redownloaded both v1 and v2 and neither works on my generations.

[vladmandic]

i may have downloaded the wrong one so it worked by accident as others from her work.

[TomerGadol]

Did you use the base 1.5 model for the original test? I did see that her example images were made with another model (Corneo's 7th Heaven) and I'm getting much better results with a different model (StyleJourney): No lora: With lora:

Still not the same as the example images on civitai but the effect of the lora is noticeable.