[Bug]: A1111 failing on a particular Lora model whilst other SD UI works

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

I'm experimenting with training my own Lora for one of my upcoming gigs and stumbled upon a weird issue.

One of my Lora models won't take any effect on A1111.
Weird thing is, it DOES work on other SD UI (namely ComfyUI).
All other Lora models of mine seem to work fine on both UIs.

Since all other Lora models that I made do work fine, I think I have basic "how to use" tutorials covered. The only difference is that the "not working" one used regularization images during training. I used Kohya for training & the sample images came out fine during training in both cases.

Could someone try my Lora models? I guess we can start from making sure that I'm not going insane. I have two for a comparable test. If you could dissect what's causing the issue, that would be much appreciated.

"working" Lora: https://drive.google.com/file/d/1fpTjoic02sXdq23bRqzhayTw0uXuNiuG/view?usp=sharing
"not working" Lora: https://drive.google.com/file/d/12IBGqpL3Po1Gj20b7vfMqaSn2_KDKRCx/view?usp=sharing

Output of the "not working" Lora model on ComfyUI (which come out random on A1111):

Cheers

Steps to reproduce the problem

Download these 2 Lora models and put the min the Lora folder of your A1111 installation.
Try the following prompt using NAI model (or any NAI-derived models): 1boy, <lora:X:1.0> where X should be the name of the Lora model
Confirm that the "working" model does produce images of a Naruto-looking kid, whereas the "not working" model produces random looking kids.

What should have happened?

Both Lora models should work on A1111 because they both worked on ComfyUI (although I don't know how).

Commit where the problem happens

b6af0a3

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--opt-sub-quad-attention --medvram --reinstall-torch --gradio-queue --xformers --reinstall-xformers --port 5000

List of extensions

Console logs

venv "D:\Projects\sd\stable-diffusion-webui-automatic1111\venv\Scripts\Python.exe"
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
Version: v1.3.1-16-gf7c683cb
Commit hash: f7c683cb2eeb1b183ba89d9069c46bcb21260aea
Installing torch and torchvision
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu118
Requirement already satisfied: torch==2.0.1 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (2.0.1+cu118)
Requirement already satisfied: torchvision==0.15.2 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (0.15.2+cu118)
Requirement already satisfied: jinja2 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torch==2.0.1) (3.1.2)
Requirement already satisfied: networkx in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torch==2.0.1) (2.8.6)
Requirement already satisfied: sympy in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torch==2.0.1) (1.11.1)
Requirement already satisfied: typing-extensions in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torch==2.0.1) (4.3.0)
Requirement already satisfied: filelock in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torch==2.0.1) (3.8.0)
Requirement already satisfied: requests in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torchvision==0.15.2) (2.25.1)
Requirement already satisfied: numpy in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torchvision==0.15.2) (1.23.5)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from torchvision==0.15.2) (9.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from jinja2->torch==2.0.1) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from requests->torchvision==0.15.2) (1.26.12)
Requirement already satisfied: certifi>=2017.4.17 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from requests->torchvision==0.15.2) (2022.9.14)
Requirement already satisfied: chardet<5,>=3.0.2 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from requests->torchvision==0.15.2) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from requests->torchvision==0.15.2) (2.10)
Requirement already satisfied: mpmath>=0.19 in d:\projects\sd\stable-diffusion-webui-automatic1111\venv\lib\site-packages (from sympy->torch==2.0.1) (1.3.0)
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the 'D:\Projects\sd\stable-diffusion-webui-automatic1111\venv\Scripts\python.exe -m pip install --upgrade pip' command.
Installing xformers
Collecting xformers==0.0.17
  Using cached xformers-0.0.17-cp310-cp310-win_amd64.whl (112.6 MB)
Installing collected packages: xformers
Successfully installed xformers-0.0.17
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the 'D:\Projects\sd\stable-diffusion-webui-automatic1111\venv\Scripts\python.exe -m pip install --upgrade pip' command.
Installing requirements

Launching Web UI with arguments: --opt-sub-quad-attention --medvram --reinstall-torch --gradio-queue --xformers --reinstall-xformers --port 5000
ControlNet v1.1.214
ControlNet v1.1.214
Loading weights [0d2eef9106] from D:\Projects\sd\stable-diffusion-webui-automatic1111\models\Stable-diffusion\camelliaV25x0.7+braV1x0.3.safetensors
Create LRU cache (max_size=16) for preprocessor results.
Create LRU cache (max_size=16) for preprocessor results.
Creating model from config: D:\Projects\sd\stable-diffusion-webui-automatic1111\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Running on local URL:  http://127.0.0.1:5000

To create a public link, set `share=True` in `launch()`.
Create LRU cache (max_size=16) for preprocessor results.
Startup time: 42.5s (import torch: 18.0s, import gradio: 1.4s, import ldm: 5.6s, other imports: 9.3s, setup codeformer: 0.7s, list SD models: 1.1s, load scripts: 4.8s, initialize extra networks: 0.3s, create ui: 0.6s, gradio launch: 0.5s).
Loading VAE weights specified in settings: D:\Projects\sd\stable-diffusion-webui-automatic1111\models\Stable-diffusion\vae-ft-mse-840000.vae.pt
Applying optimization: xformers... done.
Textual inversion embeddings loaded(0):
Model loaded in 7.4s (load weights from disk: 0.7s, create model: 1.8s, apply weights to model: 0.6s, apply half(): 0.6s, apply dtype to VAE: 0.2s, load VAE: 3.2s, load textual inversion embeddings: 0.2s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:14<00:00,  1.40it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00,  4.74it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00,  6.18it/s]



### Additional information

_No response_

AUTOMATIC1111 / stable-diffusion-webui