bug with benchmarking keras model?

[kkmann]

Hi,

I played around with the vignette

library("mlr3tuning")
learner = lrn("regr.keras", callbacks = list(cb_es(patience = 3)))
task = mlr_tasks$get("mtcars")
resampling = rsmp("holdout")
measure = msr("regr.mse")
tuner = tnr("grid_search", resolution = 2)
terminator = trm("evals", n_evals = 4)
instance = TuningInstanceSingleCrit$new(
  task = task,
  learner = learner,
  resampling = resampling,
  measure = measure,
  search_space = ps,
  terminator = terminator
)
tuner$optimize(instance)

I get the following error

INFO  [15:42:57.254] Starting to optimize 2 parameter(s) with '<OptimizerGridSearch>' and '<TerminatorEvals>' 
INFO  [15:42:57.367] Evaluating 1 configuration(s) 
INFO  [15:42:57.427] Benchmark with 1 resampling iterations 
INFO  [15:42:57.433] Applying learner 'regr.keras' on task 'mtcars' (iter 1/1) 
INFO  [15:43:00.008] Finished benchmark 
Error in as.vector(x, "list") : 
  cannot coerce type 'closure' to vector of type 'list'

here is my traceback

> traceback()
36: as.list.default(X)
35: as.list(X)
34: lapply(x, runlock, current_depth = current_depth + 1L)
33: FUN(X[[i]], ...)
32: lapply(x, runlock, current_depth = current_depth + 1L)
31: FUN(X[[i]], ...)
30: lapply(x, runlock, current_depth = current_depth + 1L)
29: FUN(X[[i]], ...)
28: lapply(x, runlock, current_depth = current_depth + 1L)
27: runlock(ans)
26: `[.data.table`(fact, , list(learner_param_vals = list(.SD$learner[[1L]]$param_set$values)), 
        keyby = "learner_hash")
25: fact[, list(learner_param_vals = list(.SD$learner[[1L]]$param_set$values)), 
        keyby = "learner_hash"]
24: .__ResultData__initialize(self = self, private = private, super = super, 
        data = data)
23: .subset2(public_bind_env, "initialize")(...)
22: ResultData$new(data)
21: .__BenchmarkResult__initialize(self = self, private = private, 
        super = super, data = data)
20: .subset2(public_bind_env, "initialize")(...)
19: BenchmarkResult$new(grid)
18: benchmark(design, store_models = self$store_models)
17: .__ObjectiveTuning__.eval_many(self = self, private = private, 
        super = super, xss = xss)
16: private$.eval_many(xss)
15: .__Objective__eval_many(self = self, private = private, super = super, 
        xss = xss)
14: self$objective$eval_many(xss_trafoed)
13: .__OptimInstance__eval_batch(self = self, private = private, 
        super = super, xdt = xdt)
12: inst$eval_batch(g$data[inds])
11: .__OptimizerGridSearch__.optimize(self = self, private = private, 
        super = super, inst = inst)
10: private$.optimize(inst)
9: doTryCatch(return(expr), name, parentenv, handler)
8: tryCatchOne(expr, names, parentenv, handlers[[1L]])
7: tryCatchList(expr, classes, parentenv, handlers)
6: tryCatch({
       private$.optimize(inst)
   }, terminated_error = function(cond) {
   })
5: optimize_default(inst, self, private)
4: .__Optimizer__optimize(self = self, private = private, super = super, 
       inst = inst)
3: private$.optimizer$optimize(inst)
2: .__TunerFromOptimizer__optimize(self = self, private = private, 
       super = super, inst = inst)
1: tuner$optimize(instance)

and my session info

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin19.5.0 (64-bit)
Running under: macOS Catalina 10.15.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /usr/local/Cellar/openblas/0.3.10_1/lib/libopenblasp-r0.3.10.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] paradox_0.4.0-9000    mlr3tuning_0.4.0.9000 mlr3keras_0.1.2       mlr3pipelines_0.2.1   mlr3_0.7.0-9000      
 [6] forcats_0.5.0         stringr_1.4.0         dplyr_1.0.1           purrr_0.3.4           readr_1.3.1          
[11] tidyr_1.1.1           tibble_3.0.3          ggplot2_3.3.2         tidyverse_1.3.0       keras_2.3.0.0        
[16] drake_7.12.6         

loaded via a namespace (and not attached):
 [1] colorspace_1.4-1   ellipsis_0.3.1     rprojroot_1.3-2    base64enc_0.1-3    fs_1.4.2          
 [6] rstudioapi_0.11    listenv_0.8.0      remotes_2.1.1      fansi_0.4.1        lubridate_1.7.9   
[11] xml2_1.3.2         codetools_0.2-16   knitr_1.29         pkgload_1.1.0      zeallot_0.1.0     
[16] jsonlite_1.7.1     broom_0.7.0        dbplyr_1.4.4       tfruns_1.4         compiler_4.0.2    
[21] httr_1.4.1         backports_1.1.10   assertthat_0.2.1   Matrix_1.2-18      cli_2.0.2         
[26] htmltools_0.5.0    prettyunits_1.1.1  tools_4.0.2        igraph_1.2.5       gtable_0.3.0      
[31] glue_1.4.2         Rcpp_1.0.5         cellranger_1.1.0   vctrs_0.3.4        xfun_0.15         
[36] globals_0.13.1     ps_1.4.0           mlr3measures_0.3.0 testthat_2.3.2     rvest_0.3.5       
[41] lifecycle_0.2.0    renv_0.11.0        devtools_2.3.0     future_1.19.1      scales_1.1.1      
[46] lgr_0.3.4          mlr3filters_0.3.0  hms_0.5.3          parallel_4.0.2     yaml_2.2.1        
[51] curl_4.3           memoise_1.1.0      reticulate_1.16    stringi_1.4.6      tensorflow_2.2.0  
[56] desc_1.2.0         checkmate_2.0.0    filelock_1.0.2     pkgbuild_1.0.8     storr_1.2.1       
[61] rlang_0.4.8        pkgconfig_2.0.3    evaluate_0.14      lattice_0.20-41    processx_3.4.4    
[66] tidyselect_1.1.0   here_0.1           magrittr_1.5       R6_2.4.1           generics_0.0.2    
[71] base64url_1.4      txtq_0.2.3         DBI_1.1.0          pillar_1.4.6       haven_2.3.1       
[76] whisker_0.4        withr_2.2.0        future.apply_1.6.0 modelr_0.1.8       crayon_1.3.4      
[81] uuid_0.1-4         utf8_1.1.4         rmarkdown_2.3      progress_1.2.2     usethis_1.6.1     
[86] grid_4.0.2         readxl_1.3.1       data.table_1.13.0  blob_1.2.1         callr_3.4.3       
[91] mlr3misc_0.6.0     bbotk_0.2.2        reprex_0.3.0       digest_0.6.25      munsell_0.5.0     
[96] sessioninfo_1.1.1 

[kkmann]

@mllg I think the problem somehow pops up when putting the ResultsData together

https://github.com/mlr-org/mlr3/blob/6a9359b9ea6948dce1b4a511f8f0343f63e20285/R/ResultData.R#L57

the data.table magic is hard to decypher

[pfistfl]

Hey @kkmann, sorry for taking so long to get back to you. Thanks for debugging this already.

The problem seems to be that mlr3 tries to as.list the python objects. See https://github.com/mlr-org/mlr3/issues/569 This bug was recently introduced when result objects where restructured.

This should be fixed once the upstream problem is fixed.

[pfistfl]

@kkmann Fixed, update data.table and this should run now!