`auto_tuner`: allow passing `store_*` flags?

[pat-s]

Currently all store_* options of class AutoTuner can only be passed when creating an AutoTuner via its constructor as the argument is not supported in auto_tuner.

• store_tuning_instance
• store_benchmark_result
• store_models

Is this indented or could these be added to auto_tuner as well?

Without the flag, extract_inner_tuning_results(<ResampleResult>) won't work.

library(mlr3verse)
#> Loading required package: mlr3
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")

at <- AutoTuner$new(
  learner = lrn("classif.rpart", cp = to_tune(1e-04, 1e-1, logscale = TRUE)),
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  terminator = trm("evals", n_evals = 5),
  tuner = tnr("random_search"),
  store_tuning_instance = TRUE
)

rr <- resample(tsk("iris"), at, rsmp("holdout"), store_models = TRUE)

extract_inner_tuning_results(rr)
#>    iteration        cp classif.ce learner_param_vals  x_domain task_id
#> 1:         1 -3.309696 0.09090909          <list[2]> <list[1]>    iris
#>             learner_id resampling_id
#> 1: classif.rpart.tuned       holdout

# fails

at <- auto_tuner(
  method = "random_search",
  learner = lrn("classif.rpart", cp = to_tune(1e-04, 1e-1, logscale = TRUE)),
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4,
  store_tuning_instance = TRUE
)
#> Error: Cannot set argument 'store_tuning_instance' for 'TunerRandomSearch' (not a constructor argument, not a parameter, not a field.

rr <- resample(tsk("iris"), at, rsmp("holdout"), store_models = TRUE)

^{Created on 2022-08-27 with reprex v2.0.2}

Session info

``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.2.1 (2022-06-23) #> os macOS Monterey 12.5.1 #> system aarch64, darwin20 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz Europe/Zurich #> date 2022-08-27 #> pandoc 2.19.2 @ /opt/homebrew/bin/ (via rmarkdown) #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date (UTC) lib source #> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.2.0) #> backports 1.4.1 2021-12-13 [1] CRAN (R 4.2.0) #> bbotk 0.5.3 2022-05-04 [1] CRAN (R 4.2.0) #> checkmate 2.1.0 2022-04-21 [1] CRAN (R 4.2.0) #> cli 3.3.0 2022-04-25 [1] CRAN (R 4.2.0) #> clue 0.3-61 2022-05-30 [1] CRAN (R 4.2.0) #> cluster 2.1.3 2022-03-28 [1] CRAN (R 4.2.0) #> clusterCrit 1.2.8 2018-07-26 [1] CRAN (R 4.2.0) #> codetools 0.2-18 2020-11-04 [3] CRAN (R 4.2.1) #> colorspace 2.0-3 2022-02-21 [1] CRAN (R 4.2.0) #> crayon 1.5.1 2022-03-26 [1] CRAN (R 4.2.0) #> data.table 1.14.2 2021-09-27 [1] CRAN (R 4.2.0) #> DBI 1.1.3 2022-06-18 [1] CRAN (R 4.2.0) #> digest 0.6.29 2021-12-01 [1] CRAN (R 4.2.0) #> dplyr 1.0.9 2022-04-28 [1] CRAN (R 4.2.0) #> evaluate 0.16 2022-08-09 [1] CRAN (R 4.2.1) #> fansi 1.0.3 2022-03-24 [1] CRAN (R 4.2.0) #> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.2.0) #> fs 1.5.2 2021-12-08 [1] CRAN (R 4.2.0) #> future 1.27.0 2022-07-22 [1] CRAN (R 4.2.0) #> future.apply 1.9.0 2022-04-25 [1] CRAN (R 4.2.0) #> generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.0) #> ggplot2 3.3.6 2022-05-03 [1] CRAN (R 4.2.0) #> globals 0.16.0 2022-08-05 [1] CRAN (R 4.2.0) #> glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.0) #> gtable 0.3.0 2019-03-25 [1] CRAN (R 4.2.0) #> highr 0.9 2021-04-16 [1] CRAN (R 4.2.0) #> htmltools 0.5.3 2022-07-18 [1] CRAN (R 4.2.0) #> knitr 1.40 2022-08-24 [1] CRAN (R 4.2.0) #> lgr 0.4.3 2021-09-16 [1] CRAN (R 4.2.0) #> lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.2.0) #> listenv 0.8.0 2019-12-05 [1] CRAN (R 4.2.0) #> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.0) #> mlr3 * 0.14.0 2022-08-11 [1] CRAN (R 4.2.1) #> mlr3cluster 0.1.3 2022-08-04 [1] Github (mlr-org/mlr3cluster@9cc3d23) #> mlr3data 0.6.1 2022-08-15 [1] CRAN (R 4.2.0) #> mlr3extralearners 0.5.45-9000 2022-08-08 [1] Github (mlr-org/mlr3extralearners@cd13aaf) #> mlr3filters 0.5.0 2022-01-25 [1] CRAN (R 4.2.0) #> mlr3fselect 0.7.1 2022-08-17 [1] Github (mlr-org/mlr3fselect@d615e86) #> mlr3learners 0.5.4 2022-08-15 [1] CRAN (R 4.2.0) #> mlr3measures 0.5.0 2022-08-05 [1] CRAN (R 4.2.0) #> mlr3misc 0.10.0 2022-01-11 [1] CRAN (R 4.2.0) #> mlr3pipelines 0.4.1 2022-05-15 [1] CRAN (R 4.2.0) #> mlr3tuning 0.13.1.9000 2022-08-09 [1] Github (mlr-org/mlr3tuning@679ab37) #> mlr3tuningspaces 0.3.0 2022-07-06 [1] Github (mlr-org/mlr3tuningspaces@a92670b) #> mlr3verse * 0.2.5 2022-05-18 [1] CRAN (R 4.2.0) #> mlr3viz 0.5.10 2022-08-15 [1] CRAN (R 4.2.0) #> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.2.0) #> palmerpenguins 0.1.1 2022-08-15 [1] CRAN (R 4.2.0) #> paradox 0.10.0 2022-08-06 [1] Github (mlr-org/paradox@1168a4c) #> parallelly 1.32.1 2022-07-21 [1] CRAN (R 4.2.0) #> pillar 1.8.1 2022-08-19 [1] CRAN (R 4.2.1) #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.0) #> purrr 0.3.4 2020-04-17 [1] CRAN (R 4.2.0) #> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.2.0) #> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.2.0) #> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.2.0) #> R.utils 2.12.0 2022-06-28 [1] CRAN (R 4.2.0) #> R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.0) #> rematch2 2.1.2 2020-05-01 [1] CRAN (R 4.2.0) #> reprex 2.0.2 2022-08-17 [1] CRAN (R 4.2.0) #> rlang 1.0.4 2022-07-12 [1] CRAN (R 4.2.0) #> rmarkdown 2.16 2022-08-24 [1] CRAN (R 4.2.0) #> rpart 4.1.16 2022-01-24 [3] CRAN (R 4.2.1) #> rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.2.0) #> scales 1.2.1 2022-08-20 [1] CRAN (R 4.2.1) #> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.2.0) #> stringi 1.7.8 2022-07-11 [1] CRAN (R 4.2.0) #> stringr 1.4.1 2022-08-20 [1] CRAN (R 4.2.0) #> styler 1.7.0.9001 2022-08-17 [1] Github (r-lib/styler@74c0951) #> tibble 3.1.8 2022-07-22 [1] CRAN (R 4.2.0) #> tidyselect 1.1.2 2022-02-21 [1] CRAN (R 4.2.0) #> utf8 1.2.2 2021-07-24 [1] CRAN (R 4.2.0) #> uuid 1.1-0 2022-04-19 [1] CRAN (R 4.2.0) #> vctrs 0.4.1 2022-04-13 [1] CRAN (R 4.2.0) #> withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.0) #> xfun 0.32 2022-08-10 [1] CRAN (R 4.2.1) #> yaml 2.3.5 2022-02-21 [1] CRAN (R 4.2.0) #> #> [1] /Users/pjs/Library/R/arm64/4.2/library #> [2] /opt/R/4.2.1-arm64/Resources/site-library #> [3] /opt/R/4.2.1-arm64/Resources/library #> #> ────────────────────────────────────────────────────────────────────────────── ```

[be-marc]

The function was intended as a simplification with reasonable defaults. However, I see no problem with adding them as parameters. It might take a few days. We just had a CRAN update.

[pat-s]

The help page only references it as "Syntactic sugar". No sentence is currently mentioning its purpose of being a simplification with reasonable defaults. Maybe in the book? But the help page should also make that clear I think.

AFAICS this would apply to multiple sugar functions (e.g. tune(), etc.). Should we be more explicit in the help page? And treat these functions as synonyms for their respective R6 class?

It might take a few days. We just had a CRAN update.

No rush, I can just use the R6 constructor for the time being.

[be-marc]

Should we be more explicit in the help page? And treat these functions as synonyms for their respective R6 class?

Yes, I'm on it. Thank you for your comments.

[be-marc]

Done.