search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

Add precision arg for pretraining #1353

Closed rasbt closed 3 weeks ago

rasbt commented 3 weeks ago

Allows selecting the precision for pretraining.

awaelchli commented 3 weeks ago

In #882 I set it to mixed because of convergence stability. It would be great if we could leave the TinyLlama defaults untouched to be consistent with the original repro settings. For pretraining, I see mixed as the better default, even for the toy examples. Can we keep that please?

rasbt commented 3 weeks ago

Arg yes, typing "bf16-true" must have been a weird muscle memory reflex. I didn't intentionally mean to change this. Thanks for the note.