Support of the LyCORIS (LoCon/LoHA) models

[mkhennoussi]

Model/Pipeline/Scheduler description

Hi everyone ! Thanks for your amazing work !

Some specific (optimized) version of Lora are developed (https://github.com/KohakuBlueleaf/LyCORIS) and are available (https://civitai.com/models/37053/michael-jordan) with pretty cool features (ability to make distinctions between trained concepts, etc.). It could be nice, as support for loading .safetensors is made, to have the possibility to load also LyCORIS models. What do you think ?

Thanks a lot !

Open source status

• [X] The model implementation is available
• [X] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[JemiloII]

I would love to see this feature as well. However, most implementations will bake it into their LoRA implementations. So maybe extending LoRAs to use these and not check metadata names vigorously would be a start.

[JemiloII]

https://github.com/huggingface/diffusers/pull/3294 there is a pr here to support it.

[carl10086]

I encountered an issue while using version 0.17.1 of the library, specifically when calling the load_lora_weights method. I received the following error:

• it's a lycoris from this url: https://civitai.com/models/76404/lycoris-couture-by-edg

ValueError("Network alpha is not consistent")
This error originates from the following method:

_convert_kohya_lora_to_diffusers(state_dict)

To bypass this error, I tried commenting out the following lines of code:

    for key, value in state_dict.items():
        if "lora_down" in key:
            lora_name = key.split(".")[0]
            lora_name_up = lora_name + ".lora_up.weight"
            lora_name_alpha = lora_name + ".alpha"
            if lora_name_alpha in state_dict:
                alpha = state_dict[lora_name_alpha].item()
                if network_alpha is None:
                    network_alpha = alpha
                # elif network_alpha != alpha:
                #     raise ValueError("Network alpha is not consistent")

The code now runs, but this is not the appropriate solution.

I would appreciate any suggestions or advice on how to properly handle this

[patrickvonplaten]

cc @sayakpaul

[sayakpaul]

I encountered an issue while using version 0.17.1 of the library, specifically when calling the load_lora_weights method. I received the following error:

• it's a lycoris from this url: https://civitai.com/models/76404/lycoris-couture-by-edg

ValueError("Network alpha is not consistent")
This error originates from the following method:

_convert_kohya_lora_to_diffusers(state_dict)

To bypass this error, I tried commenting out the following lines of code:

    for key, value in state_dict.items():
        if "lora_down" in key:
            lora_name = key.split(".")[0]
            lora_name_up = lora_name + ".lora_up.weight"
            lora_name_alpha = lora_name + ".alpha"
            if lora_name_alpha in state_dict:
                alpha = state_dict[lora_name_alpha].item()
                if network_alpha is None:
                    network_alpha = alpha
                # elif network_alpha != alpha:
                #     raise ValueError("Network alpha is not consistent")

The code now runs, but this is not the appropriate solution.

I would appreciate any suggestions or advice on how to properly handle this

Did you get the expected outputs for this? We added that check for robustness in the module. Cc: @takuma104.

There can be different configurations for LyCORIS and from the get-go, it's not possible to support all of them. So, we started supporting them minimally: https://huggingface.co/docs/diffusers/main/en/training/lora#supporting-a1111-themed-lora-checkpoints-from-diffusers.

Question for @takuma104:

Do we want to relax this constraint? https://github.com/huggingface/diffusers/blob/0bab447670f47c28df60fbd2f6a0f833f75a16f5/src/diffusers/loaders.py#L1242

I think that might break things as that would mean different alphas for different LoRA layers, no?

[carl10086]

1. I am in full agreement with your current viewpoint, which is why I have chosen to raise this issue in this particular thread concerning LyCORIS.

2. LyCORIS in a1111 does have an array of formats, making it indeed challenging to support them all at once.

3. LyCORIS getting more and more in civitai :)

[takuma104]

@sayakpaul The constraint you mention indeed assumes that all network_alphas are same, as you stated. This assumption comes from the simplification of propagating network_alpha through a single variable. To correct this, we could potentially propagate all network_alphas that correspond to each lora-weight. I believe we should incorporate this improvement when we address the LoCon revisions. From what I've gathered, some LoCon files have unique network_alphas, while others do not.

I believe the LoCon support essentially extends the methodology of #3756. Therefore, I think it would be best to first finalize #3756 as the mechanism, and then create a separate PR for LoCon support.

[sayakpaul]

I concur with your thoughts, @takuma104. https://github.com/huggingface/diffusers/pull/3778 is about to be get merged soon. So, we can start #3756 and what you have gathered pretty soon.

Cc: @patrickvonplaten

[sayakpaul]

@takuma104 went through https://gist.github.com/takuma104/dcf4626fe2b0564d02c6edd4e9fcb616 I saw either 4 or 32 for the LoRAs you have listed there. But within the same LoRA, the alpha value didn't change. Perhaps, I missed something?

[takuma104]

@sayakpaul The first comment is mostly with network_alpha=4, but there are some parts where network_alpha=1. This key is unfamiliar to me, so I think this might be the LoCon part.

lora_unet_down_blocks_0_downsamplers_0_conv.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_0_conv1.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_0_conv2.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_1_conv1.alpha 1.0 1 1.0
lora_unet_down_blocks_0_resnets_1_conv2.alpha 1.0 1 1.0

[sayakpaul]

@carl10086 we will first have support for the rest of blocks as mentioned in https://github.com/huggingface/diffusers/issues/3087#issuecomment-1602851440. Then we will revisit this as this concerns a change in how we deal with network_alphas in the LoRA layers.

Feel free to bug us in the coming weeks :)

[FurkanGozukara]

dear @sayakpaul , @patrickvonplaten

lets say i have trained a LoRA safetensors via Kohya

How can I implement it to the below pipeline?

pipe = DiffusionPipeline.from_pretrained(model_key_base, torch_dtype=torch.float16, use_auth_token=access_token)

model_dir = '/workspace'
access_token = os.getenv("ACCESS_TOKEN")

if model_dir:
    # Use local model
    model_key_base = os.path.join(model_dir, "stable-diffusion-xl-base-0.9")
    model_key_refiner = os.path.join(model_dir, "stable-diffusion-xl-refiner-0.9")
else:
    model_key_base = "stabilityai/stable-diffusion-xl-base-0.9"
    model_key_refiner = "stabilityai/stable-diffusion-xl-refiner-0.9"

# Use refiner (enabled by default)
enable_refiner = os.getenv("ENABLE_REFINER", "true").lower() == "true"
# Output images before the refiner and after the refiner
output_images_before_refiner = True

# Create public link
share = os.getenv("SHARE", "false").lower() == "true"

print("Loading model", model_key_base)
pipe = DiffusionPipeline.from_pretrained(model_key_base, torch_dtype=torch.float16, use_auth_token=access_token)

#pipe.enable_model_cpu_offload()
pipe.to("cuda")

# if using torch < 2.0
pipe.enable_xformers_memory_efficient_attention()

# pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

if enable_refiner:
    print("Loading model", model_key_refiner)
    pipe_refiner = DiffusionPipeline.from_pretrained(model_key_refiner, torch_dtype=torch.float16, use_auth_token=access_token)
    #pipe_refiner.enable_model_cpu_offload()
    pipe_refiner.to("cuda")

    # if using torch < 2.0
    pipe_refiner.enable_xformers_memory_efficient_attention()

    # pipe_refiner.unet = torch.compile(pipe_refiner.unet, mode="reduce-overhead", fullgraph=True)

# NOTE: we do not have word list filtering in this gradio demo

is_gpu_busy = False

def infer(prompt, negative, scale, samples=4, steps=50, refiner_strength=0.3, num_images=1):
    prompt, negative = [prompt] * samples, [negative] * samples
    images_b64_list = []

    for i in range(0, num_images):
        images = pipe(prompt=prompt, negative_prompt=negative, guidance_scale=scale, num_inference_steps=steps).images
        os.makedirs(r"stable-diffusion-xl-demo/outputs", exist_ok=True)
        gc.collect()
        torch.cuda.empty_cache()

        if enable_refiner:
            if output_images_before_refiner:
                for image in images:
                    buffered = BytesIO()
                    image.save(buffered, format="JPEG")
                    img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")

                    image_b64 = (f"data:image/jpeg;base64,{img_str}")
                    images_b64_list.append(image_b64)

            images = pipe_refiner(prompt=prompt, negative_prompt=negative, image=images, num_inference_steps=steps, strength=refiner_strength).images

            gc.collect()
            torch.cuda.empty_cache()

        # Create the outputs folder if it doesn't exist

        for i, image in enumerate(images):
            buffered = BytesIO()
            image.save(buffered, format="JPEG")
            img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
            timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
            image_b64 = (f"data:image/jpeg;base64,{img_str}")
            images_b64_list.append(image_b64)
            # Save the image as PNG with unique timestamp
            filename = f"stable-diffusion-xl-demo/outputs/generated_image_{timestamp}_{i}.png"
            image.save(filename, format="PNG")

    return images_b64_list

[sayakpaul]

You can follow this method:

https://huggingface.co/docs/diffusers/main/en/training/lora#supporting-a1111-themed-lora-checkpoints-from-diffusers but note that there might be incompatibilities as discussed in #3725.

[FurkanGozukara]

You can follow this method:

https://huggingface.co/docs/diffusers/main/en/training/lora#supporting-a1111-themed-lora-checkpoints-from-diffusers but note that there might be incompatibilities as discussed in #3725.

does this support SDXL?

so what is the left parameter "." ? on right we give full file path?

pipeline.load_lora_weights(".", weight_name="/workspace/light_and_shadow.safetensors")

[FurkanGozukara]

currently i am doing a LyCORIS SDXL training

would this work? @sayakpaul

pipeline.load_lora_weights(".", weight_name="/workspace/light_and_shadow.safetensors")

[sayakpaul]

If the underlying LoRA was trained against SDXL, it should work but note the following as well: https://github.com/huggingface/diffusers/issues/3725#issuecomment-1614187412

[patrickvonplaten]

Let's try to support SDXL LoRAs from the get-go :-)

[sayakpaul]

The SDXL structure is entirely different it seems and on top of that, the number of structures is large (which is already known).

[sayakpaul]

I think with https://github.com/huggingface/diffusers/pull/4147 we will have better support.

[firoz47]

Hey everyone, I know PR #4147 is in progress and will support LyCORIS/ LoCon models in future. For now is there any other way to integrate LoCon model in diffuser pipeline specifically I want to use https://civitai.com/models/47085/envybetterhands-locon model for good hands.

[sayakpaul]

You can use scripts like the one shown in: https://github.com/huggingface/diffusers/issues/3725#issue-1749079532

[sayakpaul]

Hi all!

Could you please give #4287 a try?

[alexblattner]

@sayakpaul I don't know what you're refering to exactly. Do you use the regular lora loader?

[sayakpaul]

I meant to install diffusers from the current main and giving load_lora_weights() a try with LoRA module.

[alexblattner]

What about Lycoris?

[sayakpaul]

LyCORIS LoCon is supported. LoHA is currently not. Will be soon.

[alexblattner]

With the regular Lora loader, right?

[alexblattner]

@sayakpaul will regular lora loader work with lycoris?

[sayakpaul]

load_lora_weights() should work but only for LoCon LyCORIS modules.

[ORANZINO]

Hi @sayakpaul, I'm afraid current diffusers doesn't work with LoCon for now. I've tested with my LoCon and error like below was thrown.

AttributeError: 'ModuleList' object has no attribute 'time'

Seems like there's something wrong with layer naming. Could you check this one out plz?

[sayakpaul]

Then the LoCon modules have something we don't currently support :-) IIUC, LoCon is when you apply LoRA to the conv layers as well, right? In our testing, we did consider some LoRAs that have this setup and they worked well. Check https://huggingface.co/docs/diffusers/main/en/training/lora#supporting-a1111-themed-lora-checkpoints-from-diffusers.

What is the LoRA file you're using? Could you provide a fully reproducible snippet?

[alexblattner]

@sayakpaul does locon work now?

[alexblattner]

@ORANZINO give the locon

[haofanwang]

It seems that we cannot train locon in diffusers, any plan for supporting it?

[George0726]

When I load a Lyrocis from CIVITAL https://civitai.com/models/76404/lycoris-couture-an-edg-collection, it fails,

[Darkbblue]

When I load a Lyrocis from CIVITAL https://civitai.com/models/76404/lycoris-couture-an-edg-collection, it fails,

exactly the same issue. I trained a LoCon from kohya using the sd15-EDG_LoConOptiSettings preset with the only modification to parameters being epochs.

[JustMaier]

Looks like this is supported now. This probably should be closed as complete: https://github.com/huggingface/diffusers/pull/5102

[sayakpaul]

Happily closing then :)

[Scorpinaus]

Seems like Lycoris LoCon models are supported, but the LoHA variant is not working with the latest version of diffusers (v26). Should this issue be reopened?

[sayakpaul]

If they were working with a previous version and are not working with the latest version, they, yes. Please also supply a fully reproducible snippet.