search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
7.89k stars 762 forks source link

Use trained text encoders for Flux LoRa models #1993

Open Tophness opened 5 days ago

Tophness commented 5 days ago

I trained a concept LoRa a month ago using kohya, but the trained text encoder only works with ComfyUI. The difference is night and day without it. I'm surprised there aren't any other issues about this yet. Is this feature planned?

CCpt5 commented 5 days ago

It should support this already. Did you check the console and see if any keys are ignored/mentioned as erroring? Are you setting the "Diffusion In Low Bits" to Automatic or Automatic(fp16) depending on how the LoRA was trained? If that isn't set properly the LoRA likely won't function.

In my experience training the Flux TXT encoders don't add all that much for LoRA. Haven't done enough Full Fine Tuning w/ them to have my own opinion. That said, a LoRA trained to change the TXT encoders should work properly if you don't see an error in the console about key matching.

Tophness commented 3 days ago

No keys ignored that I could see. Tried both Automatic and Automatic(fp16). For characters and objects, the unet should be the only thing that makes a real difference, but for concepts that don't already exist, the text encoder is the only thing that can make sense of them.