mlr-org / mlr3tuning

Hyperparameter optimization package of the mlr3 ecosystem
https://mlr3tuning.mlr-org.com/
GNU Lesser General Public License v3.0
53 stars 5 forks source link

Avoid storing learner with TuneToken and search space in ObjectiveTuning #286

Closed be-marc closed 3 years ago

be-marc commented 3 years ago

If a learner with TuneToken is supplied, a search space is generated in TuningInstanceSingleCrit$initialize(), TuningInstanceMultiCrit$initialize() and ~AutoTuner$initialize()~. We decided that we remove TuneTokens from learners before they are stored in ObjectiveTuning. They are not needed anymore because the information is stored in the search space. However, this is not possible if a TuneToken is stored in the values field of a parameter with the tag required. We should discus a clean solution. Currently we store the learner with the TuneToken and remove them in ObjectiveTuning.

mb706 commented 3 years ago

We could skip generating the search space in AutoTuner$initialize.

Otherwise I also made a comment about an alternative solution here.

be-marc commented 3 years ago

We could skip generating the search space in AutoTuner$initialize.

Yes we need to skip this. Otherwise we supply a learner with TuneToken and a search space to TuningInstanceSingleCrit in .train()which raises an error. This was an unrelated bug to the problem I want to discus.

Another way would be to have a "placeholder" token, similar to TuneToken, that prevents the "requires" check from triggering but has no other functionality. TuneToken could even inherit from PlaceholderToken. The semantics would be that whenever a ParamSet$values slot contains a PlaceholderToken, then its "requires" are fulfilled, its dependencies are fulfilled, but calling Learner$train() should throw an error (because the placeholder would be replaced by a concrete value in the tuning code). I.e. PlaceholderToken would be doing some part of what TuneToken is doing already, with the exception that PlaceholderToken does not generate a search_space.

_Originally posted by @mb706 in https://github.com/mlr-org/mlr3tuning/pull/285#discussion_r534472325_

Maybe we can discus this in the next call.

be-marc commented 3 years ago

A ParamSet only checks for required parameters now when $get_values() is called. Therefore, a ParamSet with missing required parameter values can exists and the described problem is solved. The ObjectiveTuning$learner slot contains a learner without TuneTuken.