Unable to load Automatic1111 LoRa due rank being higher than "min(in_features, out_features)}"

sinanhlvc commented 1 year ago

Describe the bug

Hi! I'm unable to load a LoRa as I get the following error.

ValueError: LoRA rank 384 must be less or equal than 320

The LoRa works perfectly if the ValueError check is removed from the code. Is it possible to override this error? I don't understand why it must be less or equal. Thanks!

Reproduction

Run the interference script with this LoRa Get error about the LoRa rank

Logs

  File "/home/wizard/pipelines/utils.py", line 465, in load_lora_weights_diffusers
    loader.load_lora_into_unet(state_dict, network_alphas=network_alphas, unet=unet)
  File "/home/wizard/lib/python3.9/site-packages/diffusers/loaders.py", line 1233, in load_lora_into_unet
    unet.load_attn_procs(state_dict, network_alphas=network_alphas)
  File "/home/wizardlib/python3.9/site-packages/diffusers/loaders.py", line 376, in load_attn_procs
    lora = LoRAConv2dLayer(
  File "/home/wizard/lib/python3.9/site-packages/diffusers/models/lora.py", line 58, in __init__
    raise ValueError(f"LoRA rank {rank} must be less or equal than {min(in_features, out_features)}")

System Info

diffusers version: 0.19.3
Platform: Linux-5.15.107+-x86_64-with-glibc2.31
Python version: 3.9.16
PyTorch version (GPU?): 2.0.1+cu118 (True)
Huggingface_hub version: 0.16.4
Transformers version: 4.29.2
Accelerate version: 0.16.0
xFormers version: 0.0.20
Using GPU in script?: yes
Using distributed or parallel set-up in script?: no

Who can help?

@williamberman @patrickvonplaten @saya

sayakpaul commented 1 year ago

Cc: @isidentical what do you think?

isidentical commented 1 year ago

Run the interference script with this LoRa

What is the base model you are testing with @sinanhlvc? Is it regular SD 1.5 or something else?

sinanhlvc commented 1 year ago

Run the interference script with this LoRa

What is the base model you are testing with @sinanhlvc? Is it regular SD 1.5 or something else?

Hi! Sorry I should've mentioned it. I have tested on base SD 1.5 and fine-tuned versions like Dreamshaper, deliberate, etc. and got the same error with this LoRa

sayakpaul commented 1 year ago

Possible to post a fully reproducible snippet? Preferably a Colab Notebook would be really helpful.

@isidentical WDYT?

sinanhlvc commented 1 year ago

Possible to post a fully reproducible snippet? Preferably a Colab Notebook would be really helpful.

@isidentical WDYT?

Here's the notebook.

isidentical commented 1 year ago

Looking at both A1111 and Kohya-SS's implementation, I think this is an allowed operation (where the LoRA rank outmatches the existing dimensions) (for both conv and linear): https://github.com/huggingface/diffusers/blob/aaef41b5fe06535fe6dd0caa723386c0e7494cec/src/diffusers/models/lora.py#L57-L58

E.g. this is A1111s:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/68f336bd994bed5442ad95bad6b6ad5564a5409a/extensions-builtin/Lora/network_lora.py#L35-L55

As long as the different shapes don't create a problem (not super sure whether torch or diffusers automatically reshapes somewhere else), we might be able to drop this check @sayakpaul.

williamberman commented 1 year ago

Yes this check shouldn't be necessary as the only requirement is the shapes hook together like

in_dim -> rank -> rank -> out_dim |------down----| |--------up-----|

Though if rank is larger than both in dim and out dim technically the inner projection isn't a down projection anymore :P and you lose the nice properties of the auxiliary lora blocks being smaller than the original network.

huggingface / diffusers