2-stage acquisition optimization support

[apaleyes]

This issue surfaced from conversation with @henrymoss .

Emukit does not support 2-stage optimization procedures. For example, consider the following scenario: a multifidelity function f(x,z) where z is the fidelity, that is being optimized the following way: x = argmax acq1(x) z = argmax acq2(x*, z)

A couple of multifidelity methods based on UCB are using this, and it is quite common to compare against them, so having their implementations might be useful.

[apaleyes]

@henrymoss can you link a couple of those papers please?

[henrymoss]

These three consider multi-fidelity extensions of UCB that rely on a two-stage acquisition function:

• https://arxiv.org/pdf/1703.06240.pdf
• https://arxiv.org/pdf/1603.06288.pdf
• https://arxiv.org/pdf/1811.00755.pdf

Also the FASTCV example of https://papers.nips.cc/paper/5086-multi-task-bayesian-optimization.pdf has 2-stage acquisition function.