Open teebarjunk opened 1 year ago
Tried this aswell, no success here.
If this is relevant for you: For "kohya-trainer" google colab:
!pip install -U prodigyopt
5.3
-5.5
:
optimizer_args
set the following (you can change these if you know what are you doing): ["decouple=True", "weight_decay=0.01", "d_coef=2", "use_bias_correction=True", "safeguard_warmup=True"]
unet_lr
, text_encoder_lr
to 1 (important)LoRA/config
. In config_file.toml
:
lr
(learning rates) are set to (probably) 0.1 to 1, or not set at all (otherwise error, needs checking) https://github.com/bmaltais/kohya_ss/issues/1160#issuecomment-1631606123optimizer_args
is presentoptimizer_type
is Prodigy
(safecheck)Then you have to change a bit in the kohya-trainer/train_network.py
from prodigyopt import Prodigy
#
) containing text optimizer_name, optimizer_args, optimizer = …
(Line number ~224-227)optimizer_name = "Prodigy"
optimizer_args = "['decouple': True, 'weight_decay': 0.01, 'd_coef': 2, 'use_bias_correction': True, 'safeguard_warmup': True]"
optimizer = Prodigy(trainable_params)
(first two are for metadata printing only though)
Then training should run fine (I hope)
If you re-run steps 5.2-5.4, config file will be replaced, so better to change config files directly.
Tested but I get overfitting results so early, I don't know how to set Prodigy correctly…
Tested but I get overfitting results so early, I don't know how to set Prodigy correctly…
This guide says to use more epochs than repeats, and that he gets good results.
To whom it may concern, Prodigy works nice and fast, but can easily overcook. So epochs > repeats seems best.
In section 5.2:
Set dataset_repeats
to 2
.
In section 5.4:
Set num_epochs
to something higher like 100
Set save_n_epochs_type_value
to 10
Set optimizer_args
to ["decouple=True", "weight_decay=0.01", "d_coef=2", "use_bias_correction=True", "safeguard_warmup=False", "betas=0.9,0.999"]
And in the code for 5.4, look for sample_every_n_epochs
and set that to like 10
, so you don't waste time generating previews every single epoch.
I also found this guide to keep an eye on.
I see #242 says it's implemented, but github search can't find any evidence.
Tried installing with
!pip install -U prodigyopt
and manually inputtingProdigy
as theoptimizer_type
but it didn't work.Any one had success?