search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.95k stars 733 forks source link

QLoRA subcommand #1119

Closed carmocca closed 3 weeks ago

carmocca commented 2 months ago

qlora = lora with bnb and true precision.

I think bf16-true is fine as a default because bitsandbytes doesnt' support the old cards well (like my laptop)

carmocca commented 3 weeks ago

@rasbt Do you know if this subcommand is still necessary? Or should I close this?

rasbt commented 3 weeks ago

I think that if we go the litgpt finetune REPO_ID route, then subcommands would probably not work. It would then probably have to be litgpt finetune REPO_ID --method qlora. If that's correct, you can close this, but please correct me if I'm wrong.

carmocca commented 3 weeks ago

Code is magic and anything is possible, but in general, you are correct.

Note also that to support that kind of notation, a replacement for jsonargparse will be necessary because its argument introspection capabilities are not complex enough to support that notation. I tried in the past unsuccessfully.