search

pytorch / torchtune

A Native-PyTorch Library for LLM Fine-tuning
https://pytorch.org/torchtune/main/
BSD 3-Clause "New" or "Revised" License
3.87k stars 346 forks source link

Can I just get the loss on validation and test set? #1066

Closed dexterju27 closed 1 week ago

dexterju27 commented 2 months ago

Hello,

Is there a script/function that I can use to get loss on the validation set to evaluate the checkpoint that I have. Essentially this would be put model into model.eval() and get the average loss function on validation/ test set? This could be helpful to get an insight on the fine-tuning without generating the tokens.

Thanks!

ebsmothers commented 2 months ago

Hi @dexterju27, thanks for creating the issue. Currently we do not have validation loops as part of our training scripts. If you want to quickly add one for your own dataset, you can copy the training script and/or config via tune cp, modify your local version to perform a validation loop (should be similar to the train method but with eval mode, no backward, no optimizer step, etc), then run via tune run /my/local/recipe/path --config /my/local/config/path ....

We also hope to support validation loops directly in the recipe soon, but have not gotten a chance to add this logic yet. If you'd be interesting in contributing it we'd happily review a PR!

RdoubleA commented 1 week ago

Duplicated in #883