Kernel and hyperparameter learning with applications to BayesOpt

[lacerbi]

Kernel and hyperparameter learning

• Similar to KITT in terms of training here
• Train on a "grammar" of kernels (base kernels plus linear combinations or products, random samples from hyperparameter priors)
• Show this at work on regression datasets or tasks like Bayesian optimization (see below)

Application to Bayesian optimization

• See the neural diffusion process (NDP) paper for examples (note that the OpenReview version is more recent than the arXiv version, and has more BO experiments)
• BO in 3-6 D, train on a GP, test on some standard BO function (e.g., Hartmann, Ackley)
• Acquisition function? Thompson sampling (you need correlated samples), but for us it might be convenient just to use Expected improvement (either with gradient or even without)
• Also consider BO in higher dimension e.g. 20-100 (with isotropic process and an appropriate target).

[lacerbi]

For 26.4:

• Get it to run on UCI regression datasets.
• Get started on BayesOpt problems.
• Have a look at marginalized parameters (see other issue).

[manuelhaussmann]

Note: Marginalized parameters task discussed via #8

[manuelhaussmann]

TODO

• [x] single and product kernels work within the neuralprocesses library
  • switched to Wessel's kernel library due to strange bug in my gpy code
  • sum/prod kernels still work on subset (rbf, matern, linear), periodic etc are wip
• [x] Get started on BayesOpt wip: due 05-03 05-05
• [ ] BO in 3-6 dimensions on standard BO functions due 05-07
  • have a set of common functions in 3-6d
  • for now EI as sole acquisition
  • RNP can perform the task but struggles for now to get even with/improve upon GP
  • [ ] achieving optimal performance still wip
• [ ] BO in higher dimensions (20-100) due 05-10
• [ ] basic regression experiments wip
  • principle implementation is available
  • continue one BO training routine is stable

[manuelhaussmann]

BO/Pre-slim RNP

• Setup: Hartmann in 3d; RNP learns on a variety of kernels with random hyperparameters. For simplicity these results were without products/sums.
• Results: It works. Starting with a single point (the start is randomized between runs(n_run=10), but shared with the baseline), both models struggle, but RNP is more stable. Starting with five points stabilizes both and the GP can recover its lead in the mean.

Results for a second NP baseline, sum+product kernels, higher dims should be ready by Wednesday, after which the finetuning will begin as we still follow the default hyperparameter/architecture settings.

[manuelhaussmann]

Staying with EI, five initial points we have (over 10 random starts), comparing against a GP with a Matern kernel, we have

• Hartman 3d: Learning on kernel products for RNP $\to$ we can quickly recover the GP performance, (very) slightly improving upon it in the mean and improving upon our prior performance
• Hartman 6d: Set of single kernels reaches GP performance, Product of kernels improves upon GP performance (although to be fair, a large part of that is due to one terrible run for the GP) and converges (very slightly) faster than the single kernels. (The lower plot is the product even though it is missing from the title)

Notes

• [x] Adding experiments with sums of kernels is missing
• [x] Baseline comparisons are missing
• [x] Switching to CMA is prepared but numbers are missing

Status My other paper took a lot more last minute attention time than originally planned. But by the end of today it will finally have converged to a state where it only requires minor attention until the deadline and RNP becomes my main focus.

[manuelhaussmann]

Supplementary

• [x] Describe the experimental setup
• [x] Visualize fixed Matern32 behavior in BO
• [x] Visualize variable lengthscale with Matern32
• [x] Show Single set of kernels
• [ ] (if time permits 4d/5d with Rastrigin/Ackley to have two more objectives) $\to$ cancelled for deeper exploration of hartmann

[manuelhaussmann]

ArXiv

• [x] rerun everything with the correct nr of basis functions
• [x] rerun everything with updated lenghtscales (proposal $\ell \sim LN(\log(\sqrt{d_x}/4),0.5^2)$ )
• [x] Finalize the latest plots
• [x] Do a final round of proofreading to double check consistency between main paper and supplementary

[manuelhaussmann]

Rebuttal

• [ ] Explore related papers (primarily the two mentioned below) to see differences/explore improvements dl: 28th Note: This is primarily exploratory work not a strict necessity
  • [ ] check https://arxiv.org/abs/2302.11533 for this purpose
  • they sample the GP prior lengthscales of a Matérn-5/2 as being Gamma distributed with a 99% confidence within [0.1,0.4] (instead of our log-normal) and suggest to inject a global quadratic structure to the GP trajectories to get a $f_{GP}(x) + (x - a)^TW(x-a)/d + c$, where $a,W$ are sampled at each turn and $c$ ensures a consistent range (see Sec 3.2 for further details); in terms of performance we tend to get similar regrets, just a lot slower (still have to evaluate how much of that is do to us being myopic vs the quadratic structure)
  • [ ] check https://arxiv.org/abs/2206.03992 for this purpose
  • Some details are missing from the appendix, others are contradictory. Comparing Fig 6 with ours, we are actually rather good if focus on the baselines. The main differences are (i) our {C,G}NPs fail, while their NP doesn't; (ii) their GP performs terribly (we lack all details on its structure), if ours were as bad as theirs, the curves would look at lot more similar; (iii) they don't use our ES in BO, but rather stick to picking the point with a maximal EI from a random set of 128 test locations
• [ ] Setup internal readme so that variations can easily be run during the rebuttal if needed dl: 28th (as I will have to be offline for most of the initial rebuttal time, and am only online again in the discussion period)