search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.81k stars 154 forks source link

[Bug]: EMA update step not working? #543

Closed ppbrown closed 2 weeks ago

ppbrown commented 3 weeks ago

What happened?

Either EMA update step isnt working, or it isnt working like it should (or we need a different knob)

Im using LION, const, batch 32, LR 1e-05, with around 200,000 images. 1 epoch

I did a run with update step interval=100. I then did a run with update step = 1,000

Tensorboard shows the EMA curve as identical. both reach 0.995 after 1800 steps. Out of 20,000

I need a way to adjust the EMA curve without changing the epoch count. I THOUGHT EMA update step was supposed to be that option, but apparently not.

What did you expect would happen?

I would think the EMA decay curve should change, if you change the ema update step interval

Relevant log output

No response

Output of pip freeze

No response

ppbrown commented 2 weeks ago

hum I didnt actually understand what ema update step was.

still would like a way to change the decay curve thou