search

haofanwang / Lora-for-Diffusers

The most easy-to-understand tutorial for using LoRA (Low-Rank Adaptation) within diffusers framework for AI Generation Researchers🔥
MIT License
733 stars 45 forks source link

convert lora .bin weights to ckpt or safetensors #1

Open harrywang opened 1 year ago

harrywang commented 1 year ago

Another question: convert_lora_safetensor_to_diffusers.py converts safetensors to diffusers format. After I trained LoRA model, I have the following in the output folder and checkpoint subfolder:

Screenshot 2023-02-20 at 7 07 51 PM Screenshot 2023-02-20 at 7 07 33 PM

How to convert them into safetensors like the ones I downloaded from civitai or huggingface so that I can use this via Automatica1111?

Thanks a lot!!

harrywang commented 1 year ago

https://github.com/huggingface/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py I found this but have not tried yet. Will report back once I finish the testing.

haofanwang commented 1 year ago

If I remember correctly, webui supports both safetensor and ckpt format? But anyway, the script you are using is exact what you need!

python ./scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path xxx.ckpt --dump_path save_dir --to_safetensors

Then, you can get a safetensors file. I only test it with full model (usually of more than 2GB size, if your file is just the LoRA layers, I suggest to merge them first before converting. Hope this helps.

harrywang commented 1 year ago

Thanks! I will try and report back.

haofanwang commented 1 year ago

@harrywang I guess I have understood your need, please check this response for details.

In short, we don't support convert .bin to .safetensors now. I suggest to save the full model after loading LoRA as pipe.unet.load_attn_procs(), and specify safe_serialization when saving, instead of just LoRA weights in diffusers example code. Then you can get a full model in safetensors format.

happypuppy20 commented 1 year ago

Hi @haofanwang , I'm having the SAME issue as @harrywang . Basically, we trained using lora and got those 6 files, .bin file and a few others.

To use on Automatic1111 it seems like we need to have a different extension. reading your above response as well wasn't clearly gave me any insight how I can do it.

Hope a solution would be found :) Thanks!

haofanwang commented 1 year ago

Well, the main goal of this project is for developers using diffusers, not Automatic1111. But I try my best to help.

I'm not familiar with API of Automatic1111. But to be honest, only pytorch_lora_weights.bin saves the weights of LoRA. Other files in the folders are saved by accelerate as here. What is the exact file needed for diffusers? The folder or the single .bin file? If you need the folder, I cannot support it now, you may ask help from Automatic1111 or huggingface. But if the single .bin file can work, the good news is that I'm working on it.

@happypuppy20

happypuppy20 commented 1 year ago

True, I'm referring to Automatic1111 as it's easy for us to run many tests easily and fast. Regards devs using diffusers, maybe you can tell me cause I wasn't able to find it anywhere else, but how I can inference the model with settings used by Automatic1111 like :

prompts that include large Positive , large Negative , Sampler , CFG #, sampling #, Size 512x512

I couldn't find for the Negative or Sampler , CFG...

haofanwang commented 1 year ago

Please dive into diffusers for detailed usage. There you can find how to add negative prompt in inference.

haofanwang commented 1 year ago

Update:

Hi all, I just added a new script to support converting between .bin and .safetensors. You guys can check it out.

But as I have mentioned here, it has many limitations and may not work as expected. Anyway, you can try it first. If not work, we suggest to save the full model to safetensors directly, instead of only saving LoRA weights in .bin.

harrywang commented 1 year ago

Update:

Hi all, I just added a new script to support converting between .bin and .safetensors. You guys can check it out.

But as I have mentioned here, it has many limitations and may not work as expected. Anyway, you can try it first. If not work, we suggest to save the full model to safetensors directly, instead of only saving LoRA weights in .bin.

Sorry for the slow response - I was traveling in the last few days. Thanks for the script - I tried and the conversion worked but when I load it into Automatic1111 the lora has no effect - I am not sure what went wrong.

jndietz commented 1 year ago

I'm not really familiar with the structure of diffusers or the ckpt/safetensors format. But is it just a matter of mapping keys from one object to another? I think @haofanwang has something we can start with.

harrywang commented 1 year ago

Update: Hi all, I just added a new script to support converting between .bin and .safetensors. You guys can check it out. But as I have mentioned here, it has many limitations and may not work as expected. Anyway, you can try it first. If not work, we suggest to save the full model to safetensors directly, instead of only saving LoRA weights in .bin.

Sorry for the slow response - I was traveling in the last few days. Thanks for the script - I tried and the conversion worked but when I load it into Automatic1111 the lora has no effect - I am not sure what went wrong.

1677652044281

Hi, can you help me with this problem. I convert my lora weight with your script to safetensor, but i can not use the converted lora in webui. it show this error. can you tell me what went wrong here

you can also try my script at https://github.com/huggingface/diffusers/issues/2326#issuecomment-1447341775, which works for me.

StanislawKarnacky commented 11 months ago

@harrywang, @haofanwang sorry for bothering you, but is there any chance to convert .bin to .safetensors LoRA trained on SDXL Dreambooth? Tried 5 scripts at least, keys mismatch in all cases. Trained with: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py