search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132.26k stars 26.34k forks source link

Add sanity validation steps #31047

Closed dhruvbpai closed 3 months ago

dhruvbpai commented 3 months ago

Feature request

@RylanSchaeffer

Add support for sanity validation steps before training begins, to ensure evaluation works correctly. For inspiration, Pytorch Lightning has a validation sanity check feature which is similar to what I am requesting.

Motivation

I would like to have the option to run evaluation before training starts, either for metrics or debugging purposes.

Your contribution

I can add support for this when given the green light.

amyeroberts commented 3 months ago

cc @muellerzr

muellerzr commented 3 months ago

Yeah axolotl does this, would be good for us to have this as well.

dhruvbpai commented 3 months ago

I'd be happy to get started but I am relatively new to open-source contributions. Would you be able to provide a few pointers for how you would recommend getting started, so I can take a hack at it? @muellerzr

muellerzr commented 3 months ago

cc @SunMarc not sure how far you got into this :)

SunMarc commented 3 months ago

Hi @dhruvbpai, I tried to implement what you asked ! LMK if this is what you wanted. I didn't test it yet but feel free to do it !