search

kohya-ss / sd-scripts

Apache License 2.0
4.9k stars 822 forks source link

Implementation of new optimizer: Sophia #540

Open KaraKaraWitch opened 1 year ago

KaraKaraWitch commented 1 year ago

Github: https://github.com/Liuhong99/Sophia

According to the paper on arxiv: https://arxiv.org/pdf/2305.14342.pdf It has a 2 time speed up compared to AdamW.

sdbds commented 1 year ago

i will test it soon

sdbds commented 1 year ago

image it works but original pip pack has some bugs need to fix.

sdbds commented 1 year ago

2 time speed up and 2x GPU memory required... so i need to use batch size/2 when i use it

sdbds commented 1 year ago

541 Implemented using custom parameters