Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly

[nabenabe0928]

Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with dragonfly

dragonfly

Main points

• Mixture of previous propositions by Kandasamy (dimension decomposition by additive model, multifidelity formulation)
• additive model takes the sum of models on subsets of dimensions and the intersection of the subsets must be empty
• The multifidelity is based on this post
• exponential decay kernels to introduce the monotonic decay nature of the fidelity vs performance
• the acquisition function is a weighted sum of various acquisition functions and the weights are defined in a way that we do exp3 algorithm in MAB

Experiments

Baselines

1. random search
2. HyperOpt
3. SMAC
4. Spearmint
5. GPyOpt
6. Dragonfly
7. PDOO (parallel deterministic optimistic optimisation)

Benchmarks (mostly functions except some self-made real-world examples)

1. Branin ($d = 2$)
2. Hartmann3 ($d = 3$)
3. Park1 ($d = 4$, modified version as well)
4. Park2 ($d = 4$, modified version as well)
5. Hartmann6 ($d = 6$, modified version as well)
6. Borehole ($d = 8$, modified version as well)

Although the manuscript says the functions above, apparently they used something more (especially, I am not sure about the dimensionality)

Performance over time