[Bug]: LoRAs not working the same as A1111

[RichyRich515]

Checklist

• [X] The issue exists after disabling all extensions
• [ ] The issue exists on a clean installation of webui
• [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
• [X] The issue exists in the current version of the webui
• [X] The issue has not been reported before recently
• [ ] The issue has been reported before but has not been fixed yet

What happened?

Images generated on Automatic1111's webui and imported into webui-forge will be drastically different if a LoRA is used, non-LoRA images are fine.

Automatic1111's webui:

webui-forge:

Steps to reproduce the problem

In PNG Info tab load an image generated with a LoRA from A1111's webui. Send to txt2img and try to generate the same image in webui-forge.

What should have happened?

The images should be the same. (or very close)

What browsers do you use to access the UI ?

Mozilla Firefox, Other

Sysinfo

A1111_sysinfo.txt FORGE_sysinfo.txt

Console logs

!!!--> (Not the same log as the XYZs but just an example log) <--!!!

Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]144/144 [00:09<00:00, 22.18it/s]
Version: f0.0.10-latest-79-g66c22490
Commit hash: 66c22490c3e4d55d269408959c9a692778dc36c7
Launching Web UI with arguments: --xformers --listen --enable-insecure-extension-access --port 7861 --ckpt-dir C:/Users/Richard/Desktop/stable-diffusion-webui/models/Stable-diffusion --hypernetwork-dir C:/Users/Richard/Desktop/stable-diffusion-webui/models/hypernetworks --embeddings-dir C:/Users/Richard/Desktop/stable-diffusion-webui/embeddings --lora-dir C:/Users/Richard/Desktop/stable-diffusion-webui/models/Lora --esrgan-models-path C:/Users/Richard/Desktop/stable-diffusion-webui/models/ESRGAN --vae-dir C:/Users/Richard/Desktop/stable-diffusion-webui/models/VAE
Total VRAM 24564 MB, total RAM 65436 MB
WARNING:xformers:A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : native
VAE dtype: torch.bfloat16
Using xformers cross attention
Loading weights [5dd07a46d7] from C:/Users/Richard/Desktop/stable-diffusion-webui/models/Stable-diffusion\richyrichMix-v2.fp16.safetensors
model_type EPS
UNet ADM Dimension 0
Running on local URL:  http://0.0.0.0:7861
Using xformers attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using xformers attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['model_ema.decay', 'model_ema.num_updates'])
Loading VAE weights specified in settings: C:/Users/Richard/Desktop/stable-diffusion-webui/models/VAE\vae-ft-mse-840000-ema-pruned.ckpt
To load target model SD1ClipModel
Begin to load 1 model
Moving model(s) has taken 0.16 seconds
Model loaded in 2.5s (load weights from disk: 0.4s, forge load real models: 1.1s, load VAE: 0.4s, calculate empty prompt: 0.6s).

To create a public link, set `share=True` in `launch()`.
Startup time: 21.5s (prepare environment: 3.3s, import torch: 5.2s, import gradio: 1.3s, setup paths: 1.1s, initialize shared: 0.2s, other imports: 0.5s, list SD models: 1.9s, load scripts: 3.1s, create ui: 0.4s, gradio launch: 4.4s).
Loading VAE weights specified in settings: C:/Users/Richard/Desktop/stable-diffusion-webui/models/VAE\vae-ft-mse-840000-ema-pruned.safetensors
VAE weights loaded.
                                  To load target model BaseModel
Begin to load 1 model00:00, ?it/s]
Moving model(s) has taken 0.56 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:01<00:00, 16.15it/s]
To load target model AutoencoderKL███████████████████████████▎                         | 22/36 [00:02<00:00, 19.67it/s]
Begin to load 1 model
Upscale script freed memory successfully.
tiled upscale: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 15.35it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:02<00:00,  5.44it/s]
Loading VAE weights specified in settings: cached vae-ft-mse-840000-ema-pruned.ckpt████| 36/36 [00:06<00:00,  5.22it/s]
VAE weights loaded.
Total progress: 100%|██████████████████████████████████████████████████████████████████| 36/36 [00:07<00:00,  4.76it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 36/36 [00:07<00:00,  5.22it/s]

Additional information

This issue has been observed by multiple users (more than 5) so it is not bespoke to my SW/HW setup.

[Fandango11]

I am also having the same issue after installing and updating the Forge WebUI.

[lllyasviel]

do you have download link to a lora that is influenced

[aa956]

No sample at the moment but just an observation.

Tried the full LyCORIS trained along the lines of this example configuration on SD1.5:

[LyCORIS]
network_module = "lycoris.kohya"
network_args = [ "preset=attn-mlp", "algo=full", "train_norm=True", ]

Resulting LoRA was not changing the output significantly in sd-webui-forge, while looking already overtrained in a1111 webui.

May it be related to LyCORIS?

If the sample will be required I'll get near my desktop PC on saturday and will be able to train a quick example LyCORIS.

[Vigilence]

I would recommend checking the setting in the image. Could be that the seed settings for both programs are not set to match.

[FallenIncursio]

do you have download link to a lora that is influenced

https://civitai.com/models/139062/ring-world-background-halo-lora

Drastic changes can also be seen on this LoRA: https://civitai.com/models/9430?modelVersionId=271567

Automatic1111's webui:

webui-forge:

I also checked every setting on my auto1111 and forge, to make sure, there is no key difference.

The XL Version seems to be working fine: https://civitai.com/models/294722/tsunomaki-watame-sdxl-or-hololive webui-forge:

Automatic1111's webui:

[hako-mikan]

This is my guess, but it seems that the process of applying the patch for LoRA's TextEncoder is not being carried out. Within the scope of my testing, the results matched when I disabled LoRA's TextEncoder in the Web-UI and when I used LoRA with forge. Since many of the XL LoRAs do not learn the TextEncoder, the problem tends to be overlooked.

[lllyasviel]

update and try again?

[Fandango11]

That seems like it fixed it for me!

Thanks

[RichyRich515]

Works now, thank you!

webui-forge @ g54edd297

masterpiece, best quality, no humans, scenery, vista, <lora:haloring-halo-richy-v1:1> bghaloring
Negative prompt: (worst quality, low quality, lowres), blurry, bokeh, depth of field, error, censored, bar censor, text, speech bubble, artist name, signature, border, sketch
Steps: 24, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1, Size: 512x768, Model hash: 5dd07a46d7, Model: richyrichMix-v2.fp16, VAE hash: 735e4c3a44, VAE: vae-ft-mse-840000-ema-pruned.ckpt, Clip skip: 2, RNG: CPU, Script: X/Y/Z plot, X Type: Seed, X Values: "1,2,3", Fixed X Values: "1, 2, 3", Y Type: Prompt S/R, Y Values: "<lora:haloring-halo-richy-v1:1>,\"\"", Version: f0.0.10-latest-94-g54edd297

[mykeehu]

Interesting. I just generated it with the same prompt, same settings in Auto1111 and Forge with lora model, and it gave me a completely different person than if I don't load the lora from the custom path (and the preview image doesn't update when I load it). As if the Forge had not loaded the Lora.

In Forge:

In Auto1111 (this good):

Update1: I solved with --xformers switch to webui-user.bat. And with xformers working now.

[jt-michels]

I just updated my Automatic to Forge and now none of my trained IA3s are remotely resembling their training subjects (worked before, and showed resemblance in kohya inference during training)... Is this an IA3 issue specifically? I almost exclusively use IA3 because of their lightweight efficiency (i.e. 1.2 MB for a full SDXL IA3 vs 200 MB for a LoRA)

I just installed forge so I am on latest release