Same setting as before, but after update(?) all loss is nan

Linaqruf / kohya-trainer

Adapted from https://note.com/kohya_ss/n/nbf7ce8d80f29 for easier cloning

Apache License 2.0

1.84k stars 304 forks source link

Same setting as before, but after update(?) all loss is nan #167

Open Raz0rStorm opened 1 year ago

Raz0rStorm commented 1 year ago

Used to be using just fine, yet today all my attempt trainning got me loss=nan Here is my config, if that should help dataset_config.txt config_file.txt

Linaqruf commented 1 year ago

Hi, I'm sorry for the late reply, I was fixing things this 2 weeks.

Personally, I do not recommend using the DAdaptation optimizer because we do not yet know the best configuration for DAdaptation, at least as far as I know.

What I can suggest is if you still want to use DAdapt, try adding ["decouple=True", "weight_decay=0.6"] to network_args.

I recommend using a newer Colab notebook because there is a new option called `min_snr_gammea, which can help maintain and reduce loss.