Extract best model from tuning instance

[phisanti]

Following this example:

library(data.table)

# define search space
search_space = ps(
  cp = p_dbl(lower = 0.001, upper = 0.1),
  minsplit = p_int(lower = 1, upper = 10)
)

# initialize instance
instance = TuningInstanceMultiCrit$new(
  task = tsk("iris"),
  learner = lrn("classif.rpart"),
  resampling = rsmp("holdout"),
  measure = msrs(c("classif.ce", "classif.acc")),
  search_space = search_space,
  terminator = trm("evals", n_evals = 5)
)

# generate design
design = data.table(cp = c(0.05, 0.01), minsplit = c(5, 3))

# eval design
instance$eval_batch(design)

Is there a way to extract the best model from the instance and predict directly on a new dataset?

[be-marc]

Two things you need to keep in mind.

1. The result is a Pareto front i.e. you will get multiple best hyperparameter configurations.
2. You can extract the models from the holdout validation but they are only fitted on 2/3 of the data set.

Usually, you want to fit on the complete data set and then predict on new data. Do you still want to extract these models? I have to admit that extracting these models is not straightforward but it is possible.

In the single-crit case, this is covered by the AutoTuner. Unfortunately, we cannot provide an AutoTuner for multi-crit optimization.

[phisanti]

My goal is to manually implement some kind of "nested cross validation" with custom sampling. I have built a function to get "combinatorial purged k-folds" (see this article) that will run within the inner loop and then go for an out of sample test. I found that I could get it via:

  resamples_res <- instance$archive$resample_result(n)
  min_res <- resamples_res$score(msr("classif.bacc")) %$% which.max(classif.bacc)
  model <- resamples_res$learners[[min_res]]
  model$train(resamples_res$task)

However, I still see a huge disparity between CV accuracy and out-of-sample test, which makes me think my implementation might not reflect the description of tune_nested() as in the mlr3 book.

[be-marc]

Are you doing single-crit or multi-crit optimization?

[phisanti]

Currently, I run a for loop with a Multi-Crit optim, but to select the best model I just use one metrict

[be-marc]

I am unsure what you want to achieve. Your code selects the model with the highest accuracy from a single ResampleResult. What value do you choose for n? In your first example, a holdout validation is used i.e. there is only one model in the ResampleResult.

When you want to select the best hyperparameter configuration based on one metric from a multi-crit archive, you do it like this.

as.data.table(instance$archive)[which.min(classif.ce), ]

The data.table contains a resample result and you can extract the model from it.

[be-marc]

Please reopen if you have further questions.