search

xichen-fy / Fira

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?
Apache License 2.0
84 stars 2 forks source link

Support Unsloth? #2

Open fzyzcjy opened 3 weeks ago

fzyzcjy commented 3 weeks ago

Hi thanks for the library! https://github.com/unslothai/unsloth is a library that supports fast PEFT fine-tuning. Therefore, I wonder whether this is / will be compatible with that?

xichen-fy commented 3 weeks ago

Thank you for your question, I will check and reply soon.

fzyzcjy @.***>于2024年10月26日 周六13:02写道：

Hi thanks for the library! https://github.com/unslothai/unsloth is a library that supports fast PEFT fine-tuning. Therefore, I wonder whether this is / will be compatible with that?

— Reply to this email directly, view it on GitHub https://github.com/xichen-fy/Fira/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/BLY3UD4HO6KJFH5BJQINONDZ5MO6XAVCNFSM6AAAAABQUOIF7SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYYTKNJVGAZTANA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

xichen-fy commented 3 weeks ago

In principle, Fira is compatible with Unsloth by directly replacing the original optimizer. We will give it a try. If you have any other good suggestions, feel free to share with us.

fzyzcjy commented 3 weeks ago

Looks great and thank you!

Btw, I am interested in, for example, fine-tuning (SFT) llama 3.1 8B on a single RTX4090 (24GB) with math datasets. Is this supported by Fira? It seems that the paper does not talk about math or (complicated) reasoning, so I wonder it is because this approach does not work for math, or because it is simply not tested yet. (I believe it is the latter; but in case it is the former I ask this)

xichen-fy commented 3 weeks ago

Thanks for your advice. Due to the DDL, we haven't evaluated our method on math reasoning datasets, and we will complete it soon.

fzyzcjy commented 3 weeks ago

Thanks for the information!