search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
7.48k stars 723 forks source link

Lora started producing blurry images after this commit #1807

Open morganavr opened 1 week ago

morganavr commented 1 week ago

On f82029c5cf6367a1070e33832b31d2c988c6c6aa commit LoRa works fine. But on 388b70134b10a71d8666a6c1c2749d7ad896467f commit and all commits after it LoRa produces blurry low quality images.

I can't pinpoint a specific commit between those two because other commits (for example 891e355fc8030503f1fb04304b869f175189b03e or 13d6f8ed900b0857e9872e67befb02f7ed54da35 ) return an error:

RuntimeError: The size of tensor a (18432) must match the size of tensor b (1769472) at non-singleton dimension 0
The size of tensor a (18432) must match the size of tensor b (1769472) at non-singleton dimension 0

7z archive with LoRa and example generation image is here: https://filebin.net/nv8kbtma944bd3j7

I trained this LoRa here: https://fal.ai/models/fal-ai/flux-lora-general-training

morganavr commented 1 week ago

Below is a difference between two commits mentioned in first 2 sentences - a seed and all other settings were the same.

image image

DocShotgun commented 1 week ago

I had this happen to me when I tried to use my lora on a Q8 GGUF flux. Fixed it by changing Diffusion in low bits to Automatic (fp16 LoRA) instead of Automatic.

morganavr commented 1 week ago

I had this happen to me when I tried to use my lora on a Q8 GGUF flux. Fixed it by changing Diffusion in low bits to Automatic (fp16 LoRA) instead of Automatic.

Thanks, it helped. "Automatic (fp16 LoRa)" is not a default option unfortunately and many users may experience the same problem with their LoRas until it's fixed somehow.

Jonseed commented 6 days ago

I'm having this exact issue with a lora I just trained, and Flux Q8 GGUF. If I change to Flux NF4 or FP8, it looks fine. So it seems there is something going on with using loras in Q8 or GGUF.

Edit: I think I had Diffusion in low bits as "Automatic." I switched it to "Automatic (fp16 LoRA)" and that fixed it.

lllyasviel commented 3 days ago

I am aware of this and may try to use better math in some updates later when I am on forge again. Someone please remind me if I forget this. Essentially baking LoRA into low bit weights is a feature somewhat only available in Forge now, and there is no perfect way to do this. Changing under hood math may make the image in this post looks better but will make some other random seeds worse. But there might be some overall better formulation for this and I will try to do some experiments

Or you can use Automatic (fp16 LoRa) to avoid baking LoRA into low bit weights, but the speed will be slower