Bug when loading LoRA with no alpha value

Eun0 commented 1 year ago

Describe the bug

I got an error message when loading LoRA in civitai using pipeline.load_lora_weights()

  File ".../python3.10/site-packages/diffusers/utils/peft_utils.py", line 147, in get_peft_kwargs
    lora_alpha = set(network_alpha_dict.values()).pop()
KeyError: 'pop from an empty set

where,

# diffusers > utils > peft_utils.py > get_peft_kwargs
if network_alpha_dict is not None: # line132
    if len(set(network_alpha_dict.values())) > 1:
        # some code
    else:
        lora_alpha = set(network_alpha_dict.values()).pop() # line 147

The LoRA model is safetensors format with no alpha value In get_peft_kwargs, it assumes the default value of network_alpha_dict is None but actual default value is empty dictionary{} after cls._convert_kohya_lora_to_diffusers() in LoraLoaderMixin.lora_state_dict()

# diffusers > loaders.py > LoraLoaderMixin > lora_state_dict
network_alphas = None
# TODO: replace it with a method from `state_dict_utils`
if all( ... ):
    # Map SDXL blocks correctly.
    if unet_config is not None:
        # use unet config to remap block numbers
        state_dict = cls._maybe_map_sgm_blocks_to_diffusers(state_dict, unet_config)
    state_dict, network_alphas = cls._convert_kohya_lora_to_diffusers(state_dict) # return dictionry

return state_dict, network_alphas

I think there are three ways to fix this bug

Change the if statement in get_peft_kwargs to check empty dictionary.

# diffusers > utils > peft_utils.py > get_peft_kwargs
######################## change ########################
if network_alpha_dict is not None and len(network_alpha_dict) > 0: 
######################## change ########################
    if len(set(network_alpha_dict.values())) > 1:
        # some code
    else:
        lora_alpha = set(network_alpha_dict.values()).pop() # line 147

Change empty dicionary to None in LoraLoaderMixin.lora_state_dict

# diffusers > loaders.py > LoraLoaderMixin > lora_state_dict
network_alphas = None 
# TODO: replace it with a method from `state_dict_utils`
if all( ... ):
    # Map SDXL blocks correctly.
    if unet_config is not None:
        # use unet config to remap block numbers
        state_dict = cls._maybe_map_sgm_blocks_to_diffusers(state_dict, unet_config)
    state_dict, network_alphas = cls._convert_kohya_lora_to_diffusers(state_dict) # return dictionry
    ######################## change ########################
    if len(network_alphas) == 0:  
        network_alphas = None     
    ######################## change ########################

return state_dict, network_alphas

Change variable name to network_alphas -> network_alphas_dict and default value as empty dictionry

# diffusers > loaders.py > LoraLoaderMixin > lora_state_dict
######################## change ########################
network_alpha_dict = {} 
######################## change ########################
# TODO: replace it with a method from `state_dict_utils`
if all( ... ):
    # Map SDXL blocks correctly.
    if unet_config is not None:
        # use unet config to remap block numbers
        state_dict = cls._maybe_map_sgm_blocks_to_diffusers(state_dict, unet_config)
    state_dict, network_alpha_dict = cls._convert_kohya_lora_to_diffusers(state_dict) ######## change 

return state_dict, network_alpha_dict ######## change

But, this way requires changing all if statements that use network_alphas parameters

Reproduction

Download ahri LoRA in (https://civitai.com/models/4789/ahri-league-of-legends-lora)

import torch
from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", dtype=torch.float16)
pipeline.load_lora_weights("weights/lora/ahri-000045.safetensors", adapter_name="ahri")

Logs

File ".../python3.10/site-packages/diffusers/utils/peft_utils.py", line 147, in get_peft_kwargs
    lora_alpha = set(network_alpha_dict.values()).pop()
KeyError: 'pop from an empty set

System Info

diffusers version: 0.22.0.dev0
Platform: Linux-4.15.0-20-generic-x86_64-with-glibc2.31
Python version: 3.10.13
PyTorch version (GPU?): 2.1.0+cu121 (True)
Huggingface_hub version: 0.17.3
Transformers version: 4.34.1
Accelerate version: 0.24.1
xFormers version: 0.0.22.post7
Using GPU in script?:
Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @patrickvonplaten

sayakpaul commented 1 year ago

Does it work when peft is not installed?

Eun0 commented 1 year ago

@sayakpaul

Yes, I got this error when updating diffusers and peft

sayakpaul commented 1 year ago

Can you check if the error goes away when peft is not installed? Maybe in a fresh environment?

Cc: @younesbelkada

Eun0 commented 1 year ago

@sayakpaul Cc: @younesbelkada

I create new conda env with

# requirements.txt
opencv-python
transformers
accelerate
xformers
git+https://github.com/huggingface/diffusers

It works

younesbelkada commented 1 year ago

Thanks for reporting and for the reproducer ! Will have a look

younesbelkada commented 1 year ago

Hi @Eun0 https://github.com/huggingface/diffusers/pull/5608 should fix your issue!

Eun0 commented 1 year ago

Hi @Eun0 #5608 should fix your issue!

Thank you!!

huggingface / diffusers