Bug: Merging lora to model with lora_add produces garbage output in stable diff webui

cloneofsimo / lora

Using Low-rank adaptation to quickly fine-tune diffusion models.

https://arxiv.org/abs/2106.09685

Apache License 2.0

6.83k stars 473 forks source link

Closed miniwa closed 1 year ago

miniwa commented 1 year ago

Steps to reproduce:

Train (or download) lora using lora_pti. I use default settings from one of the example scripts.
Merge lora with base sd-v1-5 model using lora_ add. I use: lora_add runwayml/stable-diffusion-v1-5 ./lora.safetensors ./output/sd-merged-lora.ckpt 1 --mode upl-ckpt-v2
Try to use the new model in stable diffusion webui (automatic1111).
Output is always noise/garbage.

Any ideas?

KraZZH commented 1 year ago

May I know what was the issue and how you fixed it?

miniwa commented 1 year ago

I turned out to be a hyperparameter issue, and not a model merge issue. My bad. I used the example script here: https://github.com/cloneofsimo/lora/blob/master/training_scripts/use_face_conditioning_example.sh

In particular, having scale = 8 will fry the model very quickly.