Supporting "LoRA-GA: Low-Rank Adaptation with Gradient Approximation"?

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

15.49k stars 1.04k forks source link

Supporting "LoRA-GA: Low-Rank Adaptation with Gradient Approximation"? #932

Open fzyzcjy opened 3 weeks ago

fzyzcjy commented 3 weeks ago

Hi thanks for the library! There seems to be another interesting paper: https://arxiv.org/abs/2407.05000. Thus I wonder whether Unsloth will support it?

danielhanchen commented 3 weeks ago

Interesting! Will read up on this!

fzyzcjy commented 3 weeks ago

Looking forward to it!