Can this be tested in kohya_ss

facebookresearch / schedule_free

Schedule-Free Optimization in PyTorch

Apache License 2.0

1.54k stars 49 forks source link

Can this be tested in kohya_ss #21

Closed rafstahelin closed 3 weeks ago

rafstahelin commented 2 months ago

I'd like to try this for Stable Diffusion in Kohya_ss.

cheald commented 2 months ago

All you need to do is install the library, set optimizer_type = "schedulefree.AdamWScheduleFree" in your configuration, and add optimizer.train() after the call to train_util.get_optimizer(). Works like a charm.

sdbds commented 2 months ago

https://github.com/kohya-ss/sd-scripts/pull/1250 Wait it merged into.

rafstahelin commented 1 month ago

@sdbds @cheald could you kindly suggest what basic settings to use with it? Additional parameters or optimiser args? Thanks!