search

automl / SMAC3

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
https://automl.github.io/SMAC3/v2.1.0/
Other
1.06k stars 218 forks source link

Can SMAC used to select algorithm ? #480

Closed chuchuwen closed 5 years ago

chuchuwen commented 5 years ago

Hi, I am new to hyper-parameter tuning, as I know, SMAC use bayesian optimization to tuning hyper-parameter of algorithm likes SVM, but if I have several algorithms (e.g., SVM, RandomForest, Neural Networks), can SMAC (or bayesian optimization) treat these algorithm and their hyper-parameter as a black box to optimize ? (i.e., after SMAC optimization, it can choose the best algorithm and its hyper-parameter)

mlindauer commented 5 years ago

Hi,

Thank you for your interest in SMAC. You can define a high-level categorical parameter to choose between different algorithms and define the hyperparameters for each algorithm as a conditional on this one top-level parameter.

In the end, SMAC will return to you which algorithm to use and its optimized hyperparameters.

Cheers, Marius

btwewek commented 5 years ago

Will this leads to some algorithms have not been choose during the optimization?

mlindauer commented 5 years ago

If you have a very small configuration budget and the number of your algorithms is large, yes that could happen. If your configuration budget is reasonable, the chance is quite high that each algorithm will be chosen at least once, since SMAC also interleaves configurations sampled at random.

mfeurer commented 5 years ago

For completeness I would like to point to the recent book on automated machine learning: http://www.automl.org/book/

It describes two systems (Auto-sklearn and Auto-WEKA) which both use SMAC to choose between one of the possible classification algorithms.

I'm closing this for now because the question appears to be answered. Please reopen if not.