Closed dexterju27 closed 1 week ago
Hi @dexterju27, thanks for creating the issue. Currently we do not have validation loops as part of our training scripts. If you want to quickly add one for your own dataset, you can copy the training script and/or config via tune cp
, modify your local version to perform a validation loop (should be similar to the train
method but with eval mode, no backward, no optimizer step, etc), then run via tune run /my/local/recipe/path --config /my/local/config/path ...
.
We also hope to support validation loops directly in the recipe soon, but have not gotten a chance to add this logic yet. If you'd be interesting in contributing it we'd happily review a PR!
Duplicated in #883
Hello,
Is there a script/function that I can use to get loss on the validation set to evaluate the checkpoint that I have. Essentially this would be put model into model.eval() and get the average loss function on validation/ test set? This could be helpful to get an insight on the fine-tuning without generating the tokens.
Thanks!