search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.76k stars 1.07k forks source link

QLoRA with FSDP Support #1175

Open strickvl opened 8 months ago

strickvl commented 8 months ago

I see that PEFT brought in QLoRA with FSDP support in their latest release.

Any plans to incorporate this into litgpt?

carmocca commented 8 months ago

Surely. It will need to be supported in Lightning first though. @awaelchli already had a look but there are some technical limitations to overcome