This PR allows to use continuable models in tuning. The tuner can propose models that should be continued in the objective function instead of fitting a new model.
Contract
The Tuner is solely responsible to choose models that should be continued. If the Tuner wants to continue a model, it passes the uhash of the original model and the old hyperparameter configuration with an increased budget to ObjectiveTuning.
The ObjectiveTuning can perform some extra checks before continuing a model e.g. does the learner support $continue(). These checks are always the same and therefore shouldn't be reimplemented in each tuner.
Design issue
We only allow to pass hyperparameter configuration from the tuner to the objective function (Tuner$.optimize() -> TuningInstance*$eval_batch(xdt) -> ObjectiveTuning$.eval_many(xss). Yes we allow to pass extra information in xdt but these are only stored in the archive and not available in $.eval_many(xss). xss is a list of lists of hyperparameter configurations. This design is specified by bbotk rather than mlr3tuning.
Currently, we have two issues which are caused by this design issue.
1) TunerIrace want to decide the resampling on which the hyperparameter configurations are evaluated. Currently, the resampling is more or less fixed.
2) TunerHyperband should be able to continue models. For this, the hyperparameter configurations in xss (with an increased budget) must be connected to already evaluated models in the archive.
Option 1 - Active Bindings
The Tuner can alter the objective function via active bindings. Before calling $eval_batch() the new resampling is set and a continue hash can be passed. See this PR for an implementation.
I used this method in #227 and this PR since it requires no change in bbotk. I like this solution because we can move some logic to the active bindings instead of ẃritting an increasingly long $.eval_many() method e.g. checking if the new resampling is already instantiated. However, I admit that it feels more natural to pass the hyperparameter configurations along with the corresponding resampling and continue hash in one call (Option 2).
a) The Tuner can pass resampling and continue hash along with the hyperparameter configurations. We would need to overload ObjectiveTuning$eval_many() and TuningInstance*$eval_batch() to allow these extra information to be passed to the objective function. I don't like this solution at all since we have to reimplement abstract methods from bbotk that work just fine.
b) The Tuner can pass extra information through a list to the objective function. We should change bbotk for this. Might be the most flexible solution.
instance$eval_batch(xdt, extra = list(resampling, continue_hash))
Option 3 - Config Space
We use specialized classes like TuningInstanceIrace / ObjectiveTuningIrace and TuningInstanceHyperband / ObjectiveTuningHyperand. In addition to the hyperparameter configurations, xss stores configuration parameter (resamplings and continue_hashes). We split those in $.eval_many() by having a configuration parameter set in the ObjectiveTuning* classes.
Specialized TuningInstance* classes require an AutoTuner class that switches between these classes depending on the provided tuner.
Defines the tuning scenario in clear way without secretly changing the objective in the background by the tuner.
$.eval_many() does not get bloated by code which is only useful for some tuners.
Overview
This PR allows to use continuable models in tuning. The tuner can propose models that should be continued in the objective function instead of fitting a new model.
Contract
The
Tuner
is solely responsible to choose models that should be continued. If theTuner
wants to continue a model, it passes theuhash
of the original model and the old hyperparameter configuration with an increased budget toObjectiveTuning
.The
ObjectiveTuning
can perform some extra checks before continuing a model e.g. does the learner support$continue()
. These checks are always the same and therefore shouldn't be reimplemented in each tuner.Design issue
We only allow to pass hyperparameter configuration from the tuner to the objective function (
Tuner$.optimize()
->TuningInstance*$eval_batch(xdt)
->ObjectiveTuning$.eval_many(xss)
. Yes we allow to pass extra information inxdt
but these are only stored in the archive and not available in$.eval_many(xss)
.xss
is a list of lists of hyperparameter configurations. This design is specified bybbotk
rather thanmlr3tuning
.Currently, we have two issues which are caused by this design issue.
1)
TunerIrace
want to decide the resampling on which the hyperparameter configurations are evaluated. Currently, the resampling is more or less fixed. 2)TunerHyperband
should be able to continue models. For this, the hyperparameter configurations inxss
(with an increased budget) must be connected to already evaluated models in the archive.Option 1 - Active Bindings
The
Tuner
can alter the objective function via active bindings. Before calling$eval_batch()
the new resampling is set and a continue hash can be passed. See this PR for an implementation.I used this method in #227 and this PR since it requires no change in
bbotk
. I like this solution because we can move some logic to the active bindings instead of ẃritting an increasingly long$.eval_many()
method e.g. checking if the new resampling is already instantiated. However, I admit that it feels more natural to pass the hyperparameter configurations along with the corresponding resampling and continue hash in one call (Option 2).Check out https://github.com/mlr-org/mlr3hyperband/blob/ae5ddb785cceb882fab3e64cea5a872fa0decdf4/R/TunerSuccessiveHalving.R to see what the Tuner is doing (Line 93).
Option 2 - Extras in
$eval_batch
calla) The
Tuner
can pass resampling and continue hash along with the hyperparameter configurations. We would need to overloadObjectiveTuning$eval_many()
andTuningInstance*$eval_batch()
to allow these extra information to be passed to the objective function. I don't like this solution at all since we have to reimplement abstract methods frombbotk
that work just fine.b) The
Tuner
can pass extra information through a list to the objective function. We should changebbotk
for this. Might be the most flexible solution.Option 3 - Config Space
We use specialized classes like
TuningInstanceIrace
/ObjectiveTuningIrace
andTuningInstanceHyperband
/ObjectiveTuningHyperand
. In addition to the hyperparameter configurations,xss
stores configuration parameter (resamplings and continue_hashes). We split those in$.eval_many()
by having a configuration parameter set in theObjectiveTuning*
classes.TuningInstance*
classes require anAutoTuner
class that switches between these classes depending on the provided tuner.$.eval_many()
does not get bloated by code which is only useful for some tuners.