Finetuning SDXL with TE2 set to 0, but TE2 is still training

[dwarfstar5]

Attempting a Finetune of SDXL. Unet learning rate is 5e-06, Learning rate TE1 is 1e-06, and Learning rate TE2 is set to 0….but wandb shows TE2 is training at 5e-06.

"Train text encoder" is enabled...if I disable "Train text encoder", both encoders don't train. My objective is to train TE1 and not TE2, but it doesn't seem to be working.

A7_finetune.json

[b-fission]

It looks like the gui is not passing the TE2 learn rate when it's 0. Try putting --learning_rate_te2=0 in Parameters->Advanced->Additional parameters

[dwarfstar5]

Try putting --learning_rate_te2=0 in Parameters->Advanced->Additional parameters

Thanks b-fission!...That solved it.

[ttait-vantim]

When I add that I get an error:

sdxl_train_network.py: error: unrecognized arguments: --learning_rate_te2=0

[ttait-vantim]

So it looks like sdxl_train.py has that option but not sdxl_train_network.py. I think the 1st is a fine tuning script and the second for loras? Is this not applicable to lora training as well?

[b-fission]

So it looks like sdxl_train.py has that option but not sdxl_train_network.py. I think the 1st is a fine tuning script and the second for loras? Is this not applicable to lora training as well?

Yes. sdxl_train_network.py is used for lora training, which only trains using one LR for both text encoders.

[DarkViewAI]

@b-fission are you sure lora training only does text encoder 1? and not both?

[b-fission]

@b-fission are you sure lora training only does text encoder 1? and not both?

My bad, that was typo'd when I wrote my initial reply. I've revised it now.

One way I'd check in kohya_ss is to use Utilities->Lora->Verify Lora. If an sdxl lora had been trained with text encoders, the entries for lora_te1_text_model and lora_te2_text_model would be visible in the output.

[ttait-vantim]

It prints this when starting:

create LoRA for Text Encoder 1: create LoRA for Text Encoder 2: create LoRA for Text Encoder: 264 modules. create LoRA for U-Net: 722 modules. enable LoRA for text encoder

Also found some threads on the XL pull requests that stated to use "--train_unet_only" to avoid issues with the dual text encoders...

Could that "learning_rate_te2" switch be added to the Lora script?

[b-fission]

It prints this when starting:

create LoRA for Text Encoder 1: create LoRA for Text Encoder 2: create LoRA for Text Encoder: 264 modules. create LoRA for U-Net: 722 modules. enable LoRA for text encoder

Also found some threads on the XL pull requests that stated to use "--train_unet_only" to avoid issues with the dual text encoders...

The relevant part is whether it says "enable Lora for text encoder" since "create" almost always shows up. The gui automatically uses --train_unet_only when Text Encoder learning rate is set to 0 for Lora training.

Could that "learning_rate_te2" switch be added to the Lora script?

Possibly, but that might be a question to ask at sd-scripts.