search

young-geng / EasyLM

Large language models (LLMs) made easy, EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.
Apache License 2.0
2.38k stars 254 forks source link

Add LoRA support? #58

Closed Kimiko-AI closed 1 year ago

Kimiko-AI commented 1 year ago

LoRA fine-tuning is much faster and use less memory than normal fine-tuning.

young-geng commented 1 year ago

Thanks for the suggestions! Unfortunately that's probably not on the road map of this project, since this project is mainly aiming at scaling up the training, as most of the heavy liftings in this project is designed to enable researchers to easily scale up their training to a large cluster of accelerators. These heavy liftings are mostly not necessary in the settings when LoRA is needed.