Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

Option to skip expensive final validation #1372

Open rasbt opened 2 weeks ago

rasbt commented 2 weeks ago

For quick dev and testing, the new final validation step can be very expensive, especially with larger checkpoints. This adds an option to disable it. (This is analogous to the initial_validation argument; since some people don't need the initial validation loss some times, they may also not need the final validation loss some times).

rasbt commented 2 weeks ago

This wouldn't change any default behaviors, and it would be analogous to the analogous to the initial_validation argument). Does it look ok to you @awaelchli and @carmocca or do you have any objections?

rasbt commented 1 week ago

Thanks for the feedback. In my opinion, it's worth adding it because the initial evaluation was considered too expensive, so it's optional. For the same reason, the final validation can also be sometimes too expensive when wanting to do a quick run for testing purposes (e.g. figuring out what the "Memory used" value is or if the files are saved correctly etc.). So, it's nice to have some easy control over that and be able to skip the final validation in certain cases.