To my knowledge, the various hyperparameter search methods use hyperparameter spaces defined as:
Probability distributions over HP space (random search algorithms)
Fixed sets of points in HP space (grid search, possibly Gaussian procress or tree Parzen estimators as an initial grid)
GP/TPE could also use a probability distribution initially, with some smart initial sampling scheme to pick initial points. This is what mlrMBO does. auto-sklearn provides an initial grid for GP/TPE based on hyperparameter values that work well on a library of previous datasets and calls the approach "metalearning."
So presumable we want hp_dist and hp_grid objects that both subclass hp_space objects. We could even provide semi-sane translation between the two.
hp_grid_to_dist would guess the domain of the hyperparameters
hp_dist_to_grid could sample at quantiles or on a latin space design or whatever is smartest
To specify hp_dist objects we should look at Hyperopt specifications. Doing things on log scale will probably be important, and we should think about important transformations for hyperparameters and how to handle them.
More broadly, the model/model family framework can extend beyond supervised learning. For k-means, you might want a fit.k_means_family to select k according to some reasonable strategy. Just something to keep in mind.
alexpghayes
commented
6 years ago
To my knowledge, the various hyperparameter search methods use hyperparameter spaces defined as:
GP/TPE could also use a probability distribution initially, with some smart initial sampling scheme to pick initial points. This is what
mlrMBOdoes.auto-sklearnprovides an initial grid for GP/TPE based on hyperparameter values that work well on a library of previous datasets and calls the approach "metalearning."So presumable we want
hp_distandhp_gridobjects that both subclasshp_spaceobjects. We could even provide semi-sane translation between the two.hp_grid_to_distwould guess the domain of the hyperparametershp_dist_to_gridcould sample at quantiles or on a latin space design or whatever is smartestTo specify
hp_distobjects we should look at Hyperopt specifications. Doing things on log scale will probably be important, and we should think about important transformations for hyperparameters and how to handle them.More broadly, the model/model family framework can extend beyond supervised learning. For k-means, you might want a
fit.k_means_familyto selectkaccording to some reasonable strategy. Just something to keep in mind.