Hyperparameter optimization & maintanance

[samuelbelko]

Hi! Thank you for this wonderful package. I am implementing a Trust Region Bayesian Optimization (TuRBO) algorithm during Google Summer of Code, and would like to depend on AbstractGPs. A crucial part in this algorithm is recurrent optimization of hyperparameters, that appropriately stretch the trust regions.

I saw in the docs an example, Mauna Loa, for optimizing hyperparameters and wanted to ask about the current plans for providing these functionalities directly from the package. I think that I wil need to implement it in some way in the future. Would you like me to help you out with the implementation here?

I thank you very much in advance!

[Crown421]

What do you mean with "directly from the package"? If you refer to some train(gp, likelihood) this would require prescribing and depending on some Optimization package, which might not fit with user's existing preferences. Some people like using Optimjl, others like meta packages like Optimization.jl.

Though, it could be nice to have a package extension for example for Optimization.jl, which if this package is loaded, provides some convenience functions that make the problem definition easier.

[samuelbelko]

Thanks for your reply! Yes, I was referring to work-in-progress blocks in Mauna Loa example saying that the optimize_loss utility function is planned to eventually become part of JuliaGaussianProcesses.

So if I understand correctly, support for hyperparameters is beyond the scope of AbstractGPs and is not planned to be implemented here. Would there be any interest in developing a package providing such helper utilities as a part of JuliaGaussianProcesses?

[willtebbutt]

Would there be any interest in developing a package providing such helper utilities as a part of JuliaGaussianProcesses?

Absolutely. This is something that has been discussed for a long time, but no one has ever gotten around to actually doing it. If you were interested in taking a stab at such a thing, I would definitely be eager to chat!

[samuelbelko]

Okay, cool thanks for your interest! I need to discuss with my mentor how we want to solve it, so based on that, I might contact you later 👍.

[Crown421]

I think that making this an extension would be great. In this case, if e.g. Optimisation.jl is loaded, these helper functions become automatically available.

[devmotion]

if e.g. Optimisation.jl is loaded, these helper functions become automatically available.

Unfortunately, that's not possible with extensions - you can't use them to define new functions that are available conditionally. You can only use them to overload existing functions. So one approach some packages use is to define empty function stubs in the package and implement them in an extension.

[Crown421]

Unfortunately, that's not possible with extensions - you can't use them to define new functions that are available conditionally. You can only use them to overload existing functions. So one approach some packages use is to define empty function stubs in the package and implement them in an extension.

Oh, I see, that is good to know.