search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.81k stars 153 forks source link

[Feat]: Cosine with hard restarts Tunable floor #546

Open ppbrown opened 2 weeks ago

ppbrown commented 2 weeks ago

Describe your use-case.

How about adding a "..." menu option for CosHard, that allows the user to set a floor, instead of going to 0? I'm not sure what type of value would be best.. maybe just a specific LR value, where if it hits THAT value, reset to full, instead of dropping all the way to virtual-0 before the reset?

What would you like to see as a solution?

LR floor value option

Have you considered alternatives? List them here.

?

Zokreb commented 2 weeks ago

Hi community, I would second this idea :)

Calamdor commented 2 weeks ago

It is possible to do with the custom learning rate scheduler. https://github.com/Nerogar/OneTrainer/wiki/Custom-Scheduler#cosine-annealing

ppbrown commented 2 weeks ago

Thanks for the tip. But... if I wanted to mess around with things like manually filling in long.code.magic.function.names like that, I wouldnt be using OneTrainer :-/