Ft enhancements - Githubissues

AnswerDotAI / fsdp_qlora

Training LLMs with QLoRA + FSDP

Apache License 2.0

1.39k stars 186 forks source link

Ft enhancements #3

Closed KeremTurgutlu closed 7 months ago

KeremTurgutlu commented 8 months ago

LR scheduler fixes, for some reason scheduler was lagging behind so adjusted the num scheduler steps.
Added adamW as optimizer.
Another mp policy with fp32 buffer types and autocast.
QLORA paper hyperparams and training commands to run LORA vs QLORA training.

Ran 3 experiments as a sanity check to compare LORA and QLORA.

https://wandb.ai/answerdotai/fsdp/reports/LORA-vs-QLORA-FSDP--Vmlldzo2NTY4Nzkz

Refactored and fixed --precision setup.