search

kohya-ss / sd-scripts

Apache License 2.0
5.33k stars 881 forks source link

Flux how to use the lora model with T5xxl trained #1723

Open tristanwqy opened 1 month ago

tristanwqy commented 1 month ago

I have a model trained with T5-xxl = True, which is a 827M model (rank=64). When I pass this model into the WebUI Forge, I see the following logs: 

[LORA] Loaded /ai_model/dev/comfyui/models/loras/flux_with_te-step00000010.safetensors for KModel-UNet with 304 keys at weight 1.0 (skipped 0 keys) with on_the_fly = False
[LORA] Loaded /ai_model/dev/comfyui/models/loras/flux_with_te-step00000010.safetensors for KModel-CLIP with 240 keys at weight 1.0 (skipped 0 keys) with on_the_fly = False

Does this mean the T5 weights are not working at all? How can I properly use this LoRA?

kohya-ss commented 1 month ago

ComfyUI seems to support LoRA with T5XXL. I'm not sure about Forge. It may be necessary to add functionality on the Forge side.

tristanwqy commented 1 month ago

ComfyUI seems to support LoRA with T5XXL. I'm not sure about Forge. It may be necessary to add functionality on the Forge side.

Why not consider saving the LoRA or the complete matrix for T5 and CLIP separately when saving? This way, they can be directly replaced during inference, making it more convenient to use. Additionally, considering the significant difference in parameter sizes between T5 and CLIP, can it be understood that training T5 is substantially more impactful than training CLIP?

AbstractEyes commented 1 month ago

It would be helpful to include the option to save the modified CLIP_L separately as well, since merging a changed clip L from a lora is kind of intrusive. I need multiple extra steps to extract and merge, and doing this autonomously would be quite convenient.