feat: Adding several acquisition functions

[miguelgondu]

Adding several acquisition functions

Currently we have Thompson Sampling and Probability of Improvement as acquisition functions. It would be ideal to include

• [x] Expected Improvement (EI).
• [ ] Upper Confidence Bound (UCB).
• [ ] Log-Probability of Improvement (logPI).
• [ ] Log-Expected Improvement (logEI).

Describe Preferred Solution

These acquisition functions could naturally fit inside the decision making module. GPJax already has the core ingredients for e.g. acquisition function maximization using L-BFGS, and BO loops using utility-driven decision making.

Additional Context

Here are some relevant references we could use for implementing these acquisition functions:

• Taking the human out of the loop: a review of BO, by Shahriari et al., in which we can find formulas for EI and UCB.
• Unexpected Improvements to Expected Improvement for Bayesian Optimization, by Ament et al. The authors describe a numerically estable way of computing logEI.

If the feature request is approved, would you be willing to submit a PR?

Yes, I am glad to contribute implementations of e.g. UCB and logEI.

[daniel-dodd]

Thanks @miguelgondu - this would make an excellent contribution!