search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.18k stars 1.01k forks source link

LoRA+ #963

Open rasbt opened 7 months ago

rasbt commented 7 months ago

Another interesting improvement idea for our LoRA implementation:

LoRA+: Efficient Low Rank Adaptation of Large Model, https://arxiv.org/abs/2402.12354

In short, they propose different learning rates for matrices A and B to enhance feature learning, resulting in 1-2% performance gains and up to 2X faster finetuning without extra computational cost.

Of course, we would have to independently confirm before we add a feature like this.

srsawant34 commented 7 months ago

That sounds intriguing @rasbt! I'm definitely interested in contributing to LoRA+ approach. Please let me know if there's any way I can get involved and contribute.

soufiane001 commented 6 months ago

@rasbt , I'm one of the authors, happy to help with any questions!

rasbt commented 6 months ago

Thanks for the interest @srsawant34 ! Sorry, we had lot of ongoing PRs and issues in the last 2 weeks for a bit of a LitGPT redesign and must have missed this. I do have a prototype implementation that I wanted to share in the next 1-2 weeks along with a tutorial.

In the meantime, if you want to open a PR, that'd be fine too, because for me I was more focused on the experiments for now and wanted to use this as part of a "How-to modify LitGPT code and run experiments" tutorial. But this shouldn't block anything. New features and contributions are always very welcome!

Also thanks for the kind offer @soufiane001 . I'll reach out in case I have any questions.

srsawant34 commented 6 months ago

Awesome @rasbt! Meanwhile, may I kindly request you to share any resources or insights you may have regarding LoRa+ in the context of implementation?