Tuning errors for basic mlr3book example

[christophM]

I can't get the rather basic example to run from the mlr3book for tuning: https://mlr3book.mlr-org.com/tuning.html

This output and error appears:

INFO  [17:43:03.125] Starting to optimize 2 parameter(s) with '<OptimizerGridSearch>' and '<TerminatorEvals>' 
INFO  [17:43:03.130] Evaluating 1 configuration(s) 
INFO  [17:43:03.142] Benchmark with 1 resampling iterations 
INFO  [17:43:03.149] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [17:43:03.160] Finished benchmark 
Error in self$archive$add_evals(xdt, xss_trafoed, ydt) : 
  Assertion on 'ydt[, self$cols_y, with = FALSE]' failed: Contains missing values (column 'classif.ce', row 1).

This is a reproducible example (basically copy-pasted from the book.

library("mlr3")                                                                                                                        
task = tsk("pima")
learner = lrn("classif.rpart")

library("paradox")
tune_ps = ParamSet$new(list(
  ParamDbl$new("cp", lower = 0.001, upper = 0.1),
  ParamInt$new("minsplit", lower = 1, upper = 10)
))
tune_ps 

hout = rsmp("holdout")
measure = msr("classif.ce")

library("mlr3tuning")

evals20 = trm("evals", n_evals = 20)

instance = TuningInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = hout,
  measure = measure,
  search_space = tune_ps,
  terminator = evals20
)

tuner = tnr("grid_search", resolution = 5)

tuner$optimize(instance)

My sessionInfo():

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=de_DE.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mlr3tuning_0.3.0 paradox_0.4.0    mlr3_0.7.0      

loaded via a namespace (and not attached):
 [1] lgr_0.3.4          codetools_0.2-16   mlr3misc_0.5.0     listenv_0.8.0     
 [5] future_1.19.1      digest_0.6.25      crayon_1.3.4       R6_2.4.1          
 [9] mlr3measures_0.3.0 backports_1.1.10   future.apply_1.6.0 uuid_0.1-4        
[13] data.table_1.13.0  rpart_4.1-15       bbotk_0.2.0        checkmate_2.0.0   
[17] tools_4.0.2        parallel_4.0.2     compiler_4.0.2     globals_0.13.0   

[be-marc]

This was a bug in mlr3 0.6.0 but you are already using 0.7.0. I cannot reproduce this. Can anybody else reproduce this with mlr3 0.7.0?

[berndbischl]

Hi Chris, I installed all cran versions (mlr3, mlr3tuning, paradox, ml3misc), and for me this runs.

What the error seems to say is that your "classif.ce" metric eval became an NA.

Analysis: 1) This really shouldn't happen on your simple example. But currently I cannot reproduce

2) There are ways to "robustify" against this, if this really happens in practice, but here it shouldnt

3) Maybe we should still make the error message better?

[berndbischl]

@be-marc whats the error-handling approach if this really happens in practice? (Chris, that's a different problem and not really your issue here....)

[berndbischl]

do we have a test for this?

[berndbischl]

@christophM also note you should really post a seed, as this bug could happen randomly

[be-marc]

@berndbischl We use a fallback learner

[berndbischl]

@berndbischl We use a fallback learner

ok, sure, that i know. maybe my question was formulated a bit badly. I was just wondering if we can "fail more gracefully" if that happens. We could use at least a better err-msg here? I am also not sure if we should continue and let the user at least access his Archive? I know we discussed this, there's pros and cons....

[christophM]

I got the same issue with a regression task and mse/mae. And consistently for repeated runs, so not at random. I will reinstall mlr3 and the other packages and see if I can get it working.

[berndbischl]

I got the same issue with a regression task and mse/mae. And consistently for repeated runs, so not at random

well, just a general hint, as your code is not "reproducible" in general without a seed.

[christophM]

I will include a seed next time, had not thought about the randomness of the tuning process.

After re-installing c("mlr3", "mlr3misc", "mlr3tuning", "paradox") it now works. I actually did the same installs yesterday with the exception of mlr3misc, so maybe that was the problem.