Tutorial: tuning over whole model spaces like with mlr::makeModelMultiplexer

[missuse]

First I'd like to share my appreciation about mlr3.

As a mlr user I was a bit skeptical about mlr3 since the whole R6 class thing was a nuisance that I had to learn in order to transition effectively.

After a few days of playing with mlr3 I must say I love it.

To the point.

I was very fond of mlr::makeModelMultiplexer and I could not find tutorials which would explain how to recreate it within mlr3 so I decided to create one. It looks like the code does what I intended, so I figured it might help other mlr3 users. Anyway the tutorial is below, if you like it and think it might be beneficial to mlr3 users you have my permission to modify it and share it any way you deem fit.

An example of tuning over whole model spaces with mlr3.

This example consists of tuning the filter method and the number of features selected, as well as the type of learner along with its hyper parameters

library(mlr3) 
library(mlr3pipelines)
library(mlr3learners)
library(mlr3tuning)
library(mlr3filters)
library(visNetwork)
library(paradox)

define filters to be used

filt_null <- mlr_pipeops$get("nop")

filt_ig <- mlr_pipeops$get("filter",
                           mlr_filters$get("information_gain"))

filt_mrmr <- mlr_pipeops$get("filter",
                             mlr_filters$get("mrmr"))

filt_names <- c("nop",
                "information_gain",
                "mrmr")

create a branch to these filters

graph <- mlr_pipeops$get("branch", filt_names, id = "branch1") %>>%
  gunion(list(
    filt_null,
    filt_ig,
    filt_mrmr
  ))

unbranch

graph <- graph %>>% 
  mlr_pipeops$get("unbranch",
                  filt_names,
                  id = "unbranch1") 

define learners to be used

rpart_lrn <- mlr_pipeops$get("learner",
                             learner = mlr_learners$get("classif.rpart"))
rpart_lrn$learner$predict_type <- "prob"

ranger_lrn <- mlr_pipeops$get("learner",
                              learner = mlr_learners$get("classif.ranger"))
ranger_lrn $learner$predict_type <- "prob"

lrn_names <- c("classif.rpart",
               "classif.ranger")

add learners to the graph

graph <- graph %>>% 
  mlr_pipeops$get("branch", lrn_names, id = "branch2") %>>% 
  gunion(list(
    rpart_lrn,
    ranger_lrn)) %>>% 
  mlr_pipeops$get("unbranch", lrn_names, id = "unbranch2") 

how does it look?

graph$plot(html = TRUE)

create a parameter set of all the things we would like tuned

ps <- ParamSet$new(list(
  ParamDbl$new("classif.rpart.cp", lower = 0, upper = 0.05),
  ParamInt$new("classif.ranger.mtry", lower = 1L, upper = 20L),
  ParamFct$new("classif.ranger.splitrule", levels = c("extratrees",
                                                      "gini")),
  ParamDbl$new("classif.ranger.sample.fraction", lower = 0.3, upper = 1),
  ParamInt$new("classif.ranger.num.trees", lower = 100L, upper = 2000L),
  ParamInt$new("classif.ranger.num.random.splits", lower = 1L, upper = 20L),
  ParamInt$new("information_gain.filter.nfeat", lower = 20L, upper = 60L),
  ParamFct$new("information_gain.type", levels = c("infogain", "symuncert")),
  ParamInt$new("mrmr.filter.nfeat", lower = 20L, upper = 60L),
  ParamFct$new("branch1.selection", levels = filt_names),
  ParamFct$new("branch2.selection", levels = lrn_names )

))

define dependencies in the parameter set

ps$add_dep("classif.rpart.cp",
           "branch2.selection", CondEqual$new("classif.rpart"))

ps$add_dep("classif.ranger.mtry",
           "branch2.selection", CondEqual$new("classif.ranger"))
ps$add_dep("classif.ranger.splitrule",
           "branch2.selection", CondEqual$new("classif.ranger"))
ps$add_dep("classif.ranger.sample.fraction",
           "branch2.selection", CondEqual$new("classif.ranger"))
ps$add_dep("classif.ranger.num.trees",
           "branch2.selection", CondEqual$new("classif.ranger"))
ps$add_dep("classif.ranger.num.random.splits",
           "classif.ranger.splitrule", CondEqual$new("extratrees"))

ps$add_dep("information_gain.filter.nfeat",
           "branch1.selection", CondEqual$new("information_gain"))
ps$add_dep("information_gain.type",
           "branch1.selection", CondEqual$new("information_gain"))
ps$add_dep("mrmr.filter.nfeat",
           "branch1.selection", CondEqual$new("mrmr"))

convert graph to learner and define resampling, measures and random search

glrn <- GraphLearner$new(graph) 

glrn$predict_type <- "prob"

cv5 <- rsmp("cv", folds = 5)

tsk <- mlr_tasks$get("sonar")

stratified sampling based on target

tsk$col_roles$stratum <- tsk$target_names

create instance and tune

instance <- TuningInstance$new(
  task = tsk,
  learner = glrn,
  resampling = cv5,
  measures = msr("classif.auc"),
  param_set = ps,
  terminator = term("evals", n_evals = 20)
)

tuner <- TunerRandomSearch$new()
tuner$tune(instance)
instance$result

Kind regards,

Milan

[berndbischl]

First I'd like to share my appreciation about mlr3. As a mlr user I was a bit skeptical about mlr3 since the whole R6 class thing was a nuisance that I had to learn in order to transition effectively. After a few days of playing with mlr3 I must say I love it.

Ok, to keep it short: Thx + great + this is how we intended it :) I have received now multiple messages ala "I am/was scared of R6 / Hey do I really have to learn this????". Can we from your perspective do anything to help people make the transition? I am a bit "blind" here because for me this does not seem hard, but I have also > 20 years of coding under my belt...

[berndbischl]

Regrading the rest: This is supercool. But: Can I please move the issu to the mlr3book? Because this is where this info should be available. As text and also as a worked out usecase. Are you even aware of the book?

[missuse]

The transition was in fact much easier than I anticipated. After a couple of hours of messing around the implemented R6 system became intuitive.

I think the main problem is that most people do not like to put effort if they are unsure of the payoff. I was thinking do I really need to learn this new syntax just to do the same things I can now with the old package. When I realized what kind of maneuvering space mlr3 offers I stopped thinking about the effort, learning mlr3 became as interesting as learning mlr when I was transitioning from caret.

I think the best way to attract users is examples. Examples of how to perform things you can do in mlr, caret and scikit-learn that are there are nice but more are needed, especially examples of complex pipelines that previously required a lot of manual code. Examples of how to integrate all the accessory packages into meaningful pipelines.

I am aware of the book, when I first checked it, about several months ago, it was in relatively early development. It covers more stuff now and it helped me understand how to make the tutorial I posted here and more. However it is in need of error checking, at least the part about pipelines has some discrepancy in code and generated plots (mentioned also here https://github.com/mlr-org/mlr3book/issues/110).

I will post the example on the book repo. I am glad you like it.

Kind regards,

Milan

[berndbischl]

thanks a lot for your feedback, and I completely agree. we are currently finetuning some technical issues with the packages, although its not a lot anymore. I am trying to gather more resources to invest them into the book, and haveing more / better usecases, maybe with videos is pretty much at the top of the list

[pat-s]

Thanks. One question that comes to my mind: Are you aware of the sugar functions (rmsp(), tsk(), flt(), etc.?). Because you were using a lot of the $get() methods.

This is important for us to know how we can push users more to the sugar side and where there is still documentation which needs to be updated.

[missuse]

I am aware of some of the sugar functions, but I figured if I am to be comfortable in using mlr3 I need to become acquainted with R6 and sugar functions just hide it.

Its something like this?

mlr_pipeops$get("filter", mlr_filters$get("information_gain"))

corresponds to

po("filter", flt("information_gain"))

and

mlr_pipeops$get("learner", learner = mlr_learners$get("classif.rpart"))

is

po("learner", lrn("classif.rpart"))

Are there any shortcuts to define dependencies in the parameter set?

[pat-s]

I am aware of some of the sugar functions, but I figured if I am to be comfortable in using mlr3 I need to become acquainted with R6 and sugar functions just hide it.

Ok thanks. Your examples are correct.

Are there any shortcuts to define dependencies in the parameter set?

Dependencies shouldn't needed to be specified from a user perspective. Are you referring to the $add_dep() method?

[missuse]

Dependencies shouldn't needed to be specified from a user perspective. Are you referring to the $add_dep() method?

Exactly. With a wider model search with more filters, learners and when also probing different data transformations I found this the most cumbersome. to define.

[jakob-r]

Exactly. With a wider model search with more filters, learners and when also probing different data transformations I found this the most cumbersome. to define.

True. And it also appears unnecessary because the dependencies are already defined through the graph. I would guess that @mb706 has some interesting thoughts on this as well. We had some discussion about the possibility to create a tuning ParmSet from a learner ParamSet. The learner ParamSet should be obtainable from the final graph including the requirements. Afterwards you should be able to subset it and set the box constraints. Do we have some functionality like that already @mb706 ?

[pfistfl]

I played around with your solution and came up with the following, slightly shorter notation:

filters = list("nop" = po("nop"),
  "information_gain" = po("filter", flt("information_gain")),
  "mrmr" = po("filter", flt("mrmr")))
learns = list("classif.rpart" = lrn("classif.rpart", predict_type = "prob"),
  "classif.ranger" = lrn("classif.ranger", predict_type = "prob"))

pipe =
  po("branch", names(filters), id = "branch1") %>>%
  gunion(unname(filters)) %>>%
  po("unbranch", names(filters), id = "unbranch1") %>>%
  po("branch", names(learns), id = "branch2") %>>%
  gunion(unname(map(learns, po, .obj = "learner"))) %>>%
  po("unbranch", names(learns), id = "unbranch2")

pipe$plot()

2. Thank you very much for the example, this shows some problems in mlr3pipelines, as the dependencies introduced by branch are currently not mirrored in the Graphs param_set. I think this is not intended @mb706 ?

3. As @jakob-r said, this shows that we definitely need for such a ParamSet construction helper when I can write down the Graph in 11 lines but need > 30 to define its ParamSet.

At a minimum, what should be possible is something like

ps$add_deps(glrn$param_set$deps))

[missuse]

I played around with your solution and came up with the following, slightly shorter notation:

Elegant, I love it. Although if you do add the example as a tutorial I suggest using a less condensed step by step form or providing a more exhaustive explanation to go with the shorter notation.

Thank you very much for the example...

You welcome, mlr3 is currently my favorite package and you can expect more feedback from me.

[mllg]

See the gallery post.