Closed rohan-gt closed 3 years ago
@rohan-gt good question! Can you clarify what you mean by "early stopping"? Do you mean:
@richardliaw to stop the hyperparameter sweep. Aren't the schedulers supported by Ray Tune used for the same purpose?
In general, we need to be able to look at some metric after each epoch to use Ray Tune's schedulers/early stopping algorithms to stop a hyperparameter sweep early. This is why we currently only early stop on estimators that have partial_fit
or warm_start
-- we can look at the metric after each epoch. Other sklearn estimators will just fit all the way to completion without giving us a chance to look at metrics in between epochs.
Hmm yeah; I think perhaps there is value to stopping the hyperparameter tuning if the top score is converged across the last X trials though (even before having fully evaluated all n_trials
trials).
@richardliaw exactly. You just need to look at the CV score progression
In the graph below I'm taking the cumulative max of the CV score as the trials progress. Here we can see that one major optimum is reached after 8 trials and we can possibly end the optimization after checking a few trials after that
Is it possible enable early stopping to any algorithm that does not have partial_fit (eg. LogisticRegression or RandomForest) just by looking at the train and test (CV score) score progression across the trials?