Is there a way to adjust the merging ratio?

[zhuofengli]

In diffusers lora documentation, you can adjust the merging ratio through cross_attention_kwargs

is there a way to do that with this safetensor approach?

If not, do you know how to convert lora safetensor to diffusers weights? I tried to use scripts that convert ckpt/safetensor to diffusers, none of them worked

[haofanwang]

(1) Yes, you can adjust the merging ratio (see here). The alpha is the ratio. I will update README.md to make it more clear.

(2) What do you mean by do that with this safetensors approach? What we have done is to insert or merge a small LoRA layer into existing models in diffusers. What is your LoRA safetensors, could you provide it and what is the error message? We have provided this example to help converting. You can also check this PR in diffusers.

If you have further questions, just pin me here.

@zhuofengli

[zhuofengli]

Thank you. alpha should be what I was looking for

My other question was do you know how to convert safetensor to diffusers weights? (Such as this one)

[haofanwang]

I see what you mean. It seems that many others (https://github.com/haofanwang/Lora-for-Diffusers/issues/1) are getting confused about different saving formats. Don't worry, Let me give more explanations here.

(1) LoRA and safetensors are two different things. LoRA is a training strategy where you usually train an extra small model, while safetensors is a saving format just like .pth or .ckpt.

(2) LoRA can be saved into different formats, which sometimes can drive us crazy. In model platforms like civitai, LoRA is saved in .safetensors; In the training example from diffusers, it is saved in .bin.

(3) Converting from .safetensors to .bin is not very common actually, and we cannot support it now. Could you describe a bit more about in what cases you need it? If you want to use it in diffusers pipeline, we don't convert .safetensors to .bin and then load via pipe.unet.load_attn_procs(), instead we merge and convert model into format like runwayml/stable-diffusion-v1-5, and you can use via pipe.from_pretrained().

(4) In summary, what we can do are as below

1. If you train your LoRA using diffusers example, you will get the new added LoRA weights in .bin. If so, diffusers has already supported to load it via pipe.unet.load_attn_procs(). No need to do any converting.
2. If you have your LoRA weights in .safetensors from other platforms (cititai), you can use our code (see Lora model only part) to merge it into existing stable-diffusion model. Then, you will get a full model in diffusers format as runwayml/stable-diffusion-v1-5. You can load it via pipe.from_pretrained()
3. If you have the full model (basemodel + LoRA) in .safetensors or .ckpt from other platforms (cititai, huggingface), you can use our code (see Full model part) to convert it to regular diffusers format as runwayml/stable-diffusion-v1-5. You can load it via pipe.from_pretrained().

@zhuofengli Please let me know if this helps.

[zhuofengli]

Thanks for the detailed explanation

What I wanted to achieve is: 1) download .safetensor from civitai 2) convert it to the format that can be used by diffusers (seems it's the .bin file) 3) at run time, use pipe.unet.load_attn_procs() to load the .bin lora file with a base model (this could be any SD based model) 4) note that we don't want to produce a merged model here, as there could be tens of loras and tens of models for user to select from

basically just trying to follow the official diffusers example here

Now I realize the approach above might not necessarily be more optimal than your convert_lora_safetensor_to_diffusers.py in this repo, which basically skips the 2nd step.

[haofanwang]

I'm trying to map the keys in safetensors to bin. I will pin you once I finish. In fact, it is very similar to the script provided. I agree this is more flexible, but not sure whether it works, anyway, I can have a shot.

[haofanwang]

@zhuofengli

Try this https://github.com/haofanwang/Lora-for-Diffusers/blob/main/format_convert.py#L161. I'm not sure whether it works, but the converting is fine. pipe.unet.load_attn_procs() only works on UNet in diffusers, while .safetensors files from civitai are usually of text encoder LoRA weights (not support in diffusers yet), for now, I just directly skip these weights. So it would be problematic if the given .safetensors contains some unsupported LoRA weights.

One more thing I wanna mention is that the naming rule is not uniform in .safetensors, thus the converting script is not generalized to all. But at least, it can give you some insights and you can speed less time to fit our need.

[ALasek]

@haofanwang Thanks for the explanation, but I'm still having issues with adjusting the weight with alpha. Some LORAs that work fine on AUTOKATIC1111's WebUi with weights just below 1 need a very low weight with your code. How does the weight scale? If I use the tag in WebUi, does it correspond to alpha=1? If I use two LORAS, let's say and , does it correspond to running your code twice sequentially with alpha=0.5?