terminator in hyperband

[pfistfl]

Just leaving this here because it cost me longer to debug than I'd care to admit:

If we set a terminator in hyperband this:

• seems to not stop executing the full bracket but instead finishes the bracket?
• stops after the bracket where e.g. number_of_evals < n_evals
• it looks like the full run finished because you get the finished optimizing printout

I guess in hyperband, we generally do not want to set a terminator. Maybe it would be good to message/warn the user in case a terminator is set with hyperband because technically we are not doing hyperband anymore when the terminator triggers?

Code, just for documentation:

library("mlr3")
library("mlr3hyperband")
library("mlr3learners")
resampling = rsmp("holdout", ratio = 0.8)
tuner = tnr("hyperband")

t = tsk("pima")$select(cols = c("pedigree", "pregnant", "age"))
l = lrn("classif.ranger")
l$param_set$values = list(
  num.trees = to_tune(p_int(lower = 1, upper = 200, tags = "budget")),
  mtry.ratio = to_tune(),
  num.threads = 1
)

at = AutoTuner$new(
  learner = l,
  resampling = resampling,
  measure = msr("classif.ce"),
  terminator = trm("evals"),
  tuner=tuner,
  store_tuning_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = TRUE
)

rr = resample(t, at, resampling, store_models = TRUE)

[be-marc]

seems to not stop executing the full bracket but instead finishes the bracket?

&

stops after the bracket where e.g. number_of_evals < n_evals

The new implementation evaluates stages with the same budget in one batch. For example, all configurations in stage 1 of bracket 3 and stage 0 of bracket 2 in one batch. We do not stop inside a batch but check the terminator after the evaluation of a complete batch. Sorry, the new documentation, which is not merged yet, explains this in more detail.

s			3			2			1			0
i		n_i	r_i		n_i	r_i		n_i	r_i		n_i	r_i
0		8	1		6	2		4	4		8	4
1		4	2		3	4		2	8
2		2	4		1	8
3		1	8

s is the bracket number, i is stage number, n_i is the number of configurations and r_i is the budget allocated to a single configuration.

Maybe it would be good to message/warn the user in case a terminator is set with hyperband because technically we are not doing hyperband anymore when the terminator triggers?

I will add a warning to the new docs.