How is hyperparameter tuning done when there is no explicit train/test set? (Cross-validation case)

[berfubuyukoz]

Hi,

I am trying to apply hyperparameter tuning on binary classification of Movie Reviews (MR) data. Since there is no explicit train/test set split for that dataset, I want to do k-fold cv like in SentEval project. But I also want to do hyperparameter optimization. I wonder how you handle this on that data (or any other data you applied cv.) In SentEval paper it says some hyperparameter tuning (such as learning rate) is performed on validation data. I would like some hints on whether you spared a custom validation data on this kind of datasets, or you did something more complex.

Thanks!

[douwekiela]

See here: https://github.com/facebookresearch/SentEval/blob/master/senteval/tools/validation.py#L57. This first finds the optimal regularization parameter using cross-validation ('optreg') and then uses it to train the final model.

[berfubuyukoz]

Thank you very much!