Open fzyzcjy opened 1 month ago
Hi thanks for the tool! I saw https://arxiv.org/abs/2402.12354 and it looks interesting (said to have better performance). Thus I wonder whether unsloth will support it?
Oh I have heard of LoRA+ - it'll require a custom optimizer like how Unsloth reduces the LR of the lm_head and embedding matrix by 10
Hi thanks for the tool! I saw https://arxiv.org/abs/2402.12354 and it looks interesting (said to have better performance). Thus I wonder whether unsloth will support it?