cloneofsimo / lora

Using Low-rank adaptation to quickly fine-tune diffusion models.
https://arxiv.org/abs/2106.09685
Apache License 2.0
6.94k stars 479 forks source link

Generated ckpt file from lora_add doesn't function correctly from training using SD 2.0+ base #201

Open xuyang06 opened 1 year ago

xuyang06 commented 1 year ago

Hi Exert,

I am using lora_pti to do fine tuning and use lora_add to generate ckpt file for webUI usage (upl-ckpt-v2 mode).

One thing looks weird to me is that all models based on SD 1.5 works well in WebUI. However, all models based on SD 2.0+ doesn't generate meaningful images, e.g., all black or white regardless of prompt input. I tried to invoke the generated safetensors files thru diffusers and they worked correctly and the generated ckpt file after invoking lora_add doesn't work well. Wondering is this known issue? Does current lora_add support generating ckpt file for SD 2.0+ models? And did we test that in SD 2.0+ before?

Thanks!

image