search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.18k stars 1.01k forks source link

QLoRA with FSDP Support #1175

Open strickvl opened 6 months ago

strickvl commented 6 months ago

I see that PEFT brought in QLoRA with FSDP support in their latest release.

Any plans to incorporate this into litgpt?

carmocca commented 6 months ago

Surely. It will need to be supported in Lightning first though. @awaelchli already had a look but there are some technical limitations to overcome