Bayesian Optimization with different Surrogate Models

[marcozanotti]

As far as I understand:

• Bayesian Optimization is available in tidymodels - tune using the function tune_bayes, and
• the tune_bayes function uses GPfit::GP_fit() to fit a Gaussian Process as surrogate model.

It would be very nice to have the possibility to set different surrogate models, especially tree-based models like Random Forest. The reason is that Gaussian Process does not work well when the parameter space is not continuous.

BTW thanks for your work!

[topepo]

Thanks for the suggestion. I don't know that we will make that change, or at least not soon. I definitely see your point about GPs and categorical tuning parameters. I think that you'll have to rely on grid search (maybe made more efficient with racing) to optimize those.

It's not a huge undertaking but it isn't obvious what we should do. BO relies on a variance measure that is driven by spatial effects and random forests and a lot of other models don't estimate anything like that. Plus what does extrapolation mean when the tuning parameters are not continuous?

I'm going to close this in a few days (see if there are more comments). However, I will be thinking of how we could do it. I'll revisit the issue in a few months (once we are over the bolus of releases and planned features we currently have). If something pops up, I'll re-open the issue.