Model not trained in tuning

[RaphaelS1]

Sorry if this isn't a bug but is expected behaviour.

The example below demonstrates that measures with requires_learner are passed the untrained learner in tuning.

library(mlr3)
library(mlr3tuning)
library(paradox)

MeasureClassifTest <- R6::R6Class("MeasureClassifTest",
  inherit = MeasureClassif,
  public = list(
    initialize = function() {
      super$initialize(
        id = "classif.test",
        range = c(-Inf, Inf),
        minimize = TRUE,
        predict_type = "response",
        properties = c("requires_learner")
      )
    }
  ),
  private = list(
    .score = function(prediction, learner, task, train_set, ...) {
      m <- learner$model
      stop(ifelse(is.null(m), "Model NULL", "Model Available"))
    }
  )
)

l <- lrn("classif.debug")
t <- tsk("iris")
l$train(t)
# returns model as expected
l$predict(t)$score(MeasureClassifTest$new(), learner = l)
#> Error in get_private(measure)$.score(prediction = as_prediction(prediction, : Model Available

ps <- ParamSet$new(
  list(
    ParamDbl$new(id = "x", lower = 0, upper = 1)
  )
)

tuner <- tnr("grid_search")

cv <- rsmp("cv", folds = 2)

at <- AutoTuner$new(
  learner = l,
  resampling = cv,
  measure = MeasureClassifTest$new(),
  search_space = ps,
  terminator = trm("evals", n_evals = 2),
  tuner = tuner,
  store_tuning_instance = TRUE
)

# model not available
at$train(t)
#> INFO  [11:04:10.970] Starting to optimize 1 parameter(s) with '<OptimizerGridSearch>' and '<TerminatorEvals>' 
#> INFO  [11:04:11.009] Evaluating 1 configuration(s) 
#> INFO  [11:04:11.048] Benchmark with 2 resampling iterations 
#> INFO  [11:04:11.302] Applying learner 'classif.debug' on task 'iris' (iter 1/2) 
#> INFO  [11:04:11.317] Applying learner 'classif.debug' on task 'iris' (iter 2/2) 
#> INFO  [11:04:11.335] Finished benchmark
#> Error in get_private(measure)$.score(prediction = as_prediction(prediction, : Model NULL

^{Created on 2021-01-14 by the reprex package (v0.3.0)}

[jakob-r]

It works with

at <- AutoTuner$new(
  learner = l,
  resampling = cv,
  measure = MeasureClassifTest$new(),
  search_space = ps,
  terminator = trm("evals", n_evals = 2),
  tuner = tuner,
  store_tuning_instance = TRUE,
  store_models = TRUE
)

However, this might be inconvenient because it will blow up the result unless you set store_benchmark_result = FALSE. Unfortunately we don't allow to have store_models = TRUE and store_benchmark_result = FALSE: https://github.com/mlr-org/mlr3tuning/blob/a8c6bf7eaaae24a5812725164ce3553eaac5c7d1/R/ObjectiveTuning.R#L57-L59 Could we relax that check and document it accordingly?

So, only if keeping all models is not an option, but you need store_benchmark_result = TRUE we have to implement something smart here: https://github.com/mlr-org/mlr3tuning/blob/a8c6bf7eaaae24a5812725164ce3553eaac5c7d1/R/ObjectiveTuning.R#L82-L83 But I would say this case can be neglected?

[RaphaelS1]

Is there a possibility to store a model up until the next iteration? So the model is kept and benchmarking is run and then the model is removed at the start of the new configuration to be tested

[jakob-r]

This would happen if you set store_models = TRUE and store_benchmark_result = FALSE which currently is not allowed.

[RaphaelS1]

Got it. Well then I hope this issue serves as a good use case to show why this may be useful to allow.

[be-marc]

We should allow store_models = TRUE and store_benchmark_result = FALSE but document that the models are only stored for one iteration and not accessible after the tuning. I am on it.

[be-marc]

Closed by 1d23f04b89ae9f88908c1430bb0ede2461127c94