Open KaraKaraWitch opened 1 year ago
Github: https://github.com/Liuhong99/Sophia
According to the paper on arxiv: https://arxiv.org/pdf/2305.14342.pdf It has a 2 time speed up compared to AdamW.
i will test it soon
it works but original pip pack has some bugs need to fix.
2 time speed up and 2x GPU memory required... so i need to use batch size/2 when i use it
Github: https://github.com/Liuhong99/Sophia
According to the paper on arxiv: https://arxiv.org/pdf/2305.14342.pdf It has a 2 time speed up compared to AdamW.