Supporting "LoRA+: Efficient Low Rank Adaptation of Large Models"

unslothai / unsloth

Finetune Llama 3.1, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

15.41k stars 1.04k forks source link

Supporting "LoRA+: Efficient Low Rank Adaptation of Large Models" #893

Open fzyzcjy opened 1 month ago

fzyzcjy commented 1 month ago

Hi thanks for the tool! I saw https://arxiv.org/abs/2402.12354 and it looks interesting (said to have better performance). Thus I wonder whether unsloth will support it?

danielhanchen commented 1 month ago

Oh I have heard of LoRA+ - it'll require a custom optimizer like how Unsloth reduces the LR of the lm_head and embedding matrix by 10