search

d8ahazard / sd_dreambooth_extension

Other
1.86k stars 282 forks source link

[Bug]: the trained LORA weights can't be loaded by SD: AssertionError:Could not find a module type (out of ModuleTypeLora,ModuleTypeHada,ModuleTypela3,ModuleTypeLokr,ModuleTypeFu 11,ModuleTypeNorm,ModuleTypeGLora,ModuleTypeOFT) #1439

Closed hanruishiren closed 3 months ago

hanruishiren commented 8 months ago

Is there an existing issue for this?

What happened?

The LORA weight trained by the latest version of dreambooth extension can't be loaded, and can't not be used by SD.

Steps to reproduce the problem

  1. I updated sdwebui(A1111) and dreambooth extension(this) to the latest version, and also the requirements. (btw: the old version of sdwebui and dreambooth extension are well runing on another PC which was built 5 month ago by me.)

  2. Then I start to train my lora as usual.

  3. During the training, everything is fine, nothing unusual.

  4. But when I test the trained LORA weights (mylor4lor4_3600.safetensors), the prompt is "mylor4lor4", the bug comes, as below.

  5. A1111 can generates images but without loading my lora weight, only loading the checkpoint weight.

Commit and libraries

python3 launch.py --listen --share --xformers --enable-insecure-extension-access --api --disable-safe-unpickle
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: 8207ccd85430fdfddcb4afa8589c88305be40f9c
Checking xformers...
Checking bitsandbytes...
Checking bitsandbytes (Linux)
Checking Dreambooth requirements...
Installed version of accelerate: 0.21.0
[Dreambooth] accelerate v0.21.0 is already installed.
Installed version of dadaptation: 3.2
[Dreambooth] dadaptation v3.2 is already installed.
Installed version of diffusers: 0.25.0
[Dreambooth] diffusers v0.25.0 is already installed.
Installed version of discord-webhook: 1.3.0
[Dreambooth] discord-webhook v1.3.0 is already installed.
Installed version of fastapi: 0.94.0
[Dreambooth] fastapi is already installed.
Installed version of gitpython: 3.1.32
[Dreambooth] gitpython v3.1.40 is not installed.
Successfully installed gitpython-3.1.41
Installed version of pytorch_optimizer: 2.12.0
[Dreambooth] pytorch_optimizer v2.12.0 is already installed.
Installed version of Pillow: 9.5.0
[Dreambooth] Pillow is already installed.
Installed version of tqdm: 4.66.1
[Dreambooth] tqdm is already installed.
Installed version of tomesd: 0.1.3
[Dreambooth] tomesd v0.1.2 is already installed.
Installed version of tensorboard: 2.13.0
[Dreambooth] tensorboard v2.13.0 is already installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.25.0 installed.
[+] bitsandbytes version 0.41.2.post2 installed.
[+] xformers version 0.0.21 installed.
Launching Web UI with arguments: --listen --share --xformers --enable-insecure-extension-access --api --disable-safe-unpickle
Style database not found: /app/stable-diffusion-webui/styles.csv
Loading weights [6ce0161689] from /app/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors
Running on local URL:  http://0.0.0.0:7860
Creating model from config: /app/stable-diffusion-webui/configs/v1-inference.yaml
Running on public URL: https://&^^%$$%&^*((&&^.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Startup time: 22.3s (prepare environment: 13.1s, import torch: 2.4s, import gradio: 0.9s, setup paths: 1.5s, initialize shared: 0.1s, other imports: 0.3s, load scripts: 1.2s, create ui: 0.7s, gradio launch: 1.7s, add APIs: 0.1s).
Applying attention optimization: xformers... done.
Model loaded in 4.1s (load weights from disk: 1.1s, create model: 0.4s, apply weights to model: 2.2s, calculate empty prompt: 0.1s).

### Command Line Arguments

```Shell
python3 launch.py --listen --share --xformers --enable-insecure-extension-access --api 

and also tried

/bin/bash webui.sh  --listen --share --xformers --enable-insecure-extension-access --api

Console logs

I guess below shows some key info, but I've no idea about it:

loading network /app/stable-diffusion-webui/models/Lora/mylor4lor4_3600.safetensors:AssertionError
Traceback (most recent call last):
File "/app/stable-diffusion-webui/extensions-builtin/Lora/networks.py",line 279,in load_networks
net load_network(name,network_on_disk)
File "/app/stable-diffusion-webui/extensions-builtin/Lora/networks.py",line 223,in load_network
raise AssertionError(f"Could not find a module type (out of {''join([x.__class__.__name__for x in module_types])3)that
would accept those keys:','join(weights.w)3")
AssertionError:Could not find a module type (out of ModuleTypeLora,ModuleTypeHada,ModuleTypela3,ModuleTypeLokr,ModuleTypeFu
11,ModuleTypeNorm,ModuleTypeGLora,ModuleTypeOFT)that would accept those keys:0.attn1.to_k.lora_down.weight,0.attn1.to_k.1o
ra_up.weight,0.attni.to_out.0.lora_down.weight,0.attn1.to_out.0.lora_up.weight,0.attn1.to_q.lora_down.weight,0.attn1.to_q.1o
ra_up.weight,0.attni.to_v.lora_down.weight,0.attni.to_v.lora_up.weight,0.attn2.to_k.lora_down.weight,0.attn2.to_k.lora_up.we
ight,0.attn2.to_out.0.lora_down.weight,0.attn2.to_out.0.lora_up.weight,0.attn2.to_q.lora_down.weight,0.attn2.to_q.lora_up.we
ight,0.attn2.to_v.lora_down.weight,0.attn2.to_v.lora_up.weight

Additional information

image

oxhak commented 8 months ago

Exact same issue here, I trained my first Lora and I get this error when I want to use it.

cryptobotany commented 8 months ago

same issue here

hanruishiren commented 8 months ago

I tested:

  1. Training the lora weight by the elder version of sdwebui and dreambooth extension, then I loaded the lora weight in the latest sdwebui and generated the images with my instance word. Everything is ok.
  2. Training the lora weight by the latest version of sdwebui and dreambooth extension, then I loaded the lora weight in the latest sdwebui, sdwebui report the error as my topic says.

Comparing these two tests, I think the problem is from dreambooth, but not from sdwebui.

FYI.

d8ahazard commented 8 months ago

What hash/revision? I'll revert

On Thu, Jan 25, 2024, 1:52 AM hanruishiren @.***> wrote:

I tested:

  1. Training the lora weight by the elder version of sdwebui and dreambooth extension, then I loaded the lora weight in the latest sdwebui and generated the images with my instance word. Everything is ok.
  2. Training the lora weight by the latest version of sdwebui and dreambooth extension, then I loaded the lora weight in the latest sdwebui, sdwebui report the error as my topic says.

Comparing these two tests, I think the problem is from dreambooth, but not from sdwebui.

FYI.

— Reply to this email directly, view it on GitHub https://github.com/d8ahazard/sd_dreambooth_extension/issues/1439#issuecomment-1909558977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NCXVXCM3HRLHBJGSKTYQIFLVAVCNFSM6AAAAABBYGCBYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBZGU2TQOJXG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cryptobotany commented 8 months ago

I ran through all the recent commit changes and the last commit that works is (731da11)f9c800ac39e0a9a668d716ce76a151453 on 21NOV23.

cryptobotany commented 8 months ago

bump

github-actions[bot] commented 7 months ago

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

Mikhael-Danilov commented 7 months ago

Same issue

github-actions[bot] commented 6 months ago

This issue is stale because it has been open for 14 days with no activity. Remove stale label or comment or this will be closed in 30 days

sharejing commented 6 months ago

Same issue

eija06 commented 6 months ago

same issue

I don't get it. The purpose of this software is to create models, but the resulting models cannot be used for months now. But this bug report again and again turns "stale", while the project is obviously active (see commits)

can somebody please help? All LORA model generations fail as shown above.

eija06 commented 6 months ago

I ran through all the recent commit changes and the last commit that works is (731da11)f9c800ac39e0a9a668d716ce76a151453 on 21NOV23.

doesn't work, same issue with this commit

eija06 commented 6 months ago

webui v1.7.0 with dreambooth extension 1.1.0 doesn't have this problem I don't recommend downgrading though, because LORA training of that version has other problems

eija06 commented 6 months ago

webui HEAD with dreambooth 1.1.0 also works this seems to be related to a change in diffusers, that dreambooth was trying to adapt to in https://github.com/d8ahazard/sd_dreambooth_extension/commit/797cb64723add661d9c009347cdb9e42b0de32c8 but fails

eija06 commented 6 months ago

The reason for this error message: AssertionError:Could not find a module type (out of ModuleTypeLora,ModuleTypeHada,ModuleTypela3,ModuleTypeLokr,ModuleTypeFu 11,ModuleTypeNorm,ModuleTypeGLora,ModuleTypeOFT)that would accept those keys:0.attn1.to_k.lora_down.weight,0.attn1.to_k.1o ra_up.weight,0.attni.to_out.0.lora_down.weight,0.attn1.to_out.0.lora_up.weight,0.attn1.to_q.lora_down.weight,0.attn1.to_q.1o ra_up.weight,0.attni.to_v.lora_down.weight,0.attni.to_v.lora_up.weight,0.attn2.to_k.lora_down.weight,0.attn2.to_k.lora_up.we ight,0.attn2.to_out.0.lora_down.weight,0.attn2.to_out.0.lora_up.weight,0.attn2.to_q.lora_down.weight,0.attn2.to_q.lora_up.we ight,0.attn2.to_v.lora_down.weight,0.attn2.to_v.lora_up.weight

is that the keys for the weights in LORAs generated by this extension are different from what sd expects.

SD expects for example

...blocks_0_attn1_to_k.lora_down.weight

while dreambooth generates files with

...blocks.0.attn1.to_k.lora_down.weight

Is anybody with knowledge of the dreambooth codebase able fix it? Should be simple enough to do now that the cause is clear. The first one is apparently how diffusers generates it, and it must be converted for SD. Which dreambooth apparently tries to do in diff_to_sd.py, but not completely.

Here is a conversion script that shows nicely the differences between diffusers and SD LORA keys, in the comments at the beginning: https://gist.github.com/eavae/a336029601f726737a0fd070f44c2f8a

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 14 days with no activity. Remove stale label or comment or this will be closed in 30 days

eija06 commented 5 months ago

this extension should be deprecated and removed from webui, if nobody takes care of major problems. this issue is not stale or closed. it wastes people's time thinking this extension still works.

github-actions[bot] commented 4 months ago

This issue is stale because it has been open for 14 days with no activity. Remove stale label or comment or this will be closed in 30 days