Is it appropriate to use halving grid search as a hyper-parameter tuning strategy?

[hengzhe-zhang]

As for the default hyper-parameter tuning strategy, I find this benchmark using the halving grid search method. To begin, I admit that the traditional grid search strategy is impractical due to the prohibitively expensive computational cost. However, when we use the halving grid search method and the parameter grid is large, as in the case of XGBoost. The first few rounds of hyper-parameter search will only train on a few data points, and the results might be unreliable. So, is this a truly good method for hyper-parameter tuning, or is this tuning protocol sufficient to persuade reviewers?

[hengzhe-zhang]

I think it is necessary to add some further explanations here. Recently, I found one paper using Bayesian optimization technique to tune the hyperparameter in their experiments [1]. This inspired me to ask this question because I want to know which method is much better. [1]. Gorishniy, Yury, et al. "Revisiting Deep Learning Models for Tabular Data." arXiv preprint arXiv:2106.11959 (2021).

[lacava]

This paper makes a good case for the appropriateness of successive halving as a hyperparameter optimization algorithm with decent guarantees: http://proceedings.mlr.press/v51/jamieson16.html. That said, I haven't run the experiment (comparing grid search with successive halving) in our context. We did use grid search for the where are we now? paper (the first SR benchmark study) and I didn't notice any large differences in performance on the overlapping problems.

I'm going to close this issue since it's more of a broad ML question, and not a research question we're trying to address here.