FEAT | Nested Cross validation

[louisgv]

🚀 Feature Proposal

In a nutshell the idea is that you can overfit a model by tuning the hyperparameters too much, so you want to have a "nested" CV approach that tells you what your level of error is on data that was not used to find the hyperparameters. If you do not use CV in the training of your model then it doesn't have to be nested, but as many models use validation error to train you often need to have a nested approach with two levels of CV

CV: Cross validation

[louisgv]

The general concept is that your model hyperparameters are still parameters. If you run the model many times to find the right hyperparameters, each of those runs is an "iteration" in the "training" of your hyperparameters. That means that, given enough iterations, you can "overfit" hyperparameters the same way that you overfit regular weights. To avoid this, you need to keep out an extra dataset that is not used in any of the training runs, so that you can measure your model's performance after hyperparameter search on data that was not used to find the optimal hyperparameters.