After a lot of experimentation (via #115), I think we should actually just use the simplest possible cross-validation that we can. Thus, this PR makes the following changes:
Uses standard v-fold CV (with 5 folds) instead of rolling-origin cross-validation. My hope is that this will increase the bias of the model slightly, since right now we seem to be low-bias, high-variance.
Replaces time-based validation sets for early stopping with random ones
Makes num_iterations, learning_rate, and max_bin tunable parameters, rather than static ones
Thus, the new splitting strategy is 90% train/10% test, then 5 folds of 80% train/20% validation, with 10% of the training set held out for early stopping.
Here's a complete CV run using the new splits. It's very similar to the time-split results, if slightly better.
After a lot of experimentation (via #115), I think we should actually just use the simplest possible cross-validation that we can. Thus, this PR makes the following changes:
num_iterations
,learning_rate
, andmax_bin
tunable parameters, rather than static onesThus, the new splitting strategy is 90% train/10% test, then 5 folds of 80% train/20% validation, with 10% of the training set held out for early stopping.
Here's a complete CV run using the new splits. It's very similar to the time-split results, if slightly better.