Mumbo

[NeoChaos12]

This is an implementation of a GP using the MUMBO acquisition function through emukit, along with some other changes in order to share functionality among all emukit based optimizers.

Upcoming: FABOLAS with MUMBO acquisition. Done, read below for update.

Some more very specific design decisions that require your opinion @KEggensperger :

1. Rather than being a Multi-Fidelity acquisition, since MUMBO is Multi-Task BO, it treats every fidelity value as a separate task. As per the example code, it also defines a separate kernel for every such task. Here, the kernels are combined such that k_i(x, x')=k_err(x, x') + ρk_i-1(x, x'), for i ϵ [2, 3, ..., N_S] and N_S is the number of individual tasks that the fidelity parameter is divided into. Are we interested in controlling the various settings related to the kernels here? If yes, how would you prefer they be exposed - a single setting for all kernels, or something different? https://github.com/automl/HPOBenchExperimentUtils/blob/d08dbb3e72959de7d8e6a2c39f00567f67c827de/HPOBenchExperimentUtils/optimizer/mumbo.py#L144-L151
2. Similar to the case above, these are also per-kernel values being set by the example code for MUMBO here. https://github.com/automl/HPOBenchExperimentUtils/blob/d08dbb3e72959de7d8e6a2c39f00567f67c827de/HPOBenchExperimentUtils/optimizer/mumbo.py#L153-L159
3. Small note on the cost estimate. I went back to check on our wrapper for dragonfly and noticed that my implementation there does add a small value of 1e-6 to prevent division by zero errors. A big difference there, though, is that since dragonfly does work on continuous fidelities, I have tried to always have a similarly linear scale of costs across fidelity types. Thus, I have now changed the cost estimation in MUMBO to be c(i) = i / N_S, effectively equivalent to dragonfly's way of handling integer fidelities (but for minor numerical differences).

Update: More notes regarding FABOLAS with MUMBO:

1. If you'd like to compare how exactly the MUMBO acquisition was inserted into FABOLAS, compare this: https://github.com/automl/HPOBenchExperimentUtils/blob/39cf1e9bcbcc57a1a86d50e2cc3f1d561d6eadba/HPOBenchExperimentUtils/optimizer/fabolas_optimizer.py#L158-L171 against this
2. The following piece of code deserves particular attention: https://github.com/automl/HPOBenchExperimentUtils/blob/39cf1e9bcbcc57a1a86d50e2cc3f1d561d6eadba/HPOBenchExperimentUtils/optimizer/fabolas_optimizer.py#L170-L171 The original code used a random search based acquisition optimizer instead, which makes little sense for MUMBO. Hence, I replaced it with the same optimization procedure as was used in vanilla MUMBO.

[NeoChaos12]

This PR is functionally quite complete. There may be some structural changes soon which can be pushed directly upstream.