jiaweizzhao / GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Apache License 2.0
1.43k stars 148 forks source link

Resume function for optimizer #35

Open bokyeong1015 opened 7 months ago

bokyeong1015 commented 7 months ago

Hi, thank you for generously open-sourcing your excellent work. During our experiments, we noticed that there doesn't seem to be a resume/reload function for the optimizer regarding args.continue_from. Is our understanding correct? If this feature has already been implemented, could you please let us know?

By the way, we found the resume function for the checkpoint model and logging information in these lines. Thank you for reviewing this inquiry.