search

mlr-org / mlr

Machine Learning in R
https://mlr.mlr-org.com
Other
1.64k stars 405 forks source link

batchmark broken #1988

Closed dagola closed 7 years ago

dagola commented 7 years ago

Using batchmark in the current version of the master results in an error when estimating the resampling error after tuning.

The error is:

Error in checkMeasures(measures, pred$task.desc, na.rm = na.rm) : 
  argument "na.rm" is missing, with no default

Function benchmark works fine.

Example

library(mlr)
library(batchtools)

# Define task ----
data(BreastCancer, package = "mlbench")
df = BreastCancer
df$Id = NULL
task = makeClassifTask(id = "BreastCancer",
                       data = df,
                       target = "Class")

# Define measures ----
measures = list(mlr::auc)

# Define filter parameter set ----
filter_ps = ParamHelpers::makeParamSet(
  ParamHelpers::makeIntegerParam("fw.abs",
                                 lower = 1,
                                 upper = mlr::getTaskNFeats(task)))

# Define tuning control ----
ctrl = mlr::makeTuneControlRandom(maxit = 10)

# Define resampling strategies ----
inner = mlr::makeResampleDesc("CV", stratify = TRUE, iters = 2)
outer = mlr::makeResampleDesc("CV", stratify = TRUE, iters = 2)

# Define random forest learner ----
rf_lrn = mlr::makeLearner("classif.ranger", predict.type = "prob", 
                          config = list(on.learner.error = "warn"))
rf_lrn = mlr::makeImputeWrapper(rf_lrn, classes = list(factor = mlr::imputeConstant("__miss__")))

rf_ps = ParamHelpers::makeParamSet(
  ParamHelpers::makeIntegerParam("num.trees", 
                                 lower = 100L, 
                                 upper = 5000L, 
                                 trafo = function(x) plyr::round_any(x, 100)),
  ParamHelpers::makeNumericParam("mtry.perc", 
                                 lower = 0.05, 
                                 upper = 0.5),
  ParamHelpers::makeIntegerParam("min.node.size",
                                 lower = 10L,
                                 upper = 100L),
  ParamHelpers::makeLogicalParam("replace"))

rf_lrn = mlr::makeTuneWrapper(rf_lrn,
                              resampling = inner,
                              control = ctrl,
                              measures = measures,
                              par.set = c(rf_ps),
                              show.info = TRUE)

bmr = mlr::benchmark(learners = list(rf_lrn), 
                      tasks = task, 
                      resamplings = outer, 
                      measures = measures, 
                      models = FALSE)

reg = batchtools::makeExperimentRegistry(packages = c("ranger"))
reg$cluster.functions = batchtools::makeClusterFunctionsInteractive()

mlr::batchmark(
  tasks = task,
  learners = list(rf_lrn),
  resamplings = outer,
  measures = measures,
  models = FALSE)

batch_ids = findNotDone()
batch_ids[, chunk := 1]
batchtools::submitJobs(ids = batch_ids)

batchtools::waitForJobs()

batchtools log

Session info

devtools::session_info()
Session info -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.3.1 (2016-06-21)
 system   x86_64, darwin16.0.0        
 ui       RStudio (1.0.143)           
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       Europe/Berlin               
 date     2017-08-23                  

Packages ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 package      * version    date       source                                   
 assertthat     0.2.0      2017-04-11 cran (@0.2.0)                            
 backports      1.1.0      2017-05-22 cran (@1.1.0)                            
 base         * 3.3.1      2016-09-30 local                                    
 base64url      1.2        2017-06-14 CRAN (R 3.3.1)                           
 batchtools   * 0.9.5-9000 2017-08-23 Github (mllg/batchtools@f1e16b9)         
 BBmisc       * 1.11       2017-08-23 Github (berndbischl/BBmisc@ca794f6)      
 bindr          0.1        2016-11-13 CRAN (R 3.3.1)                           
 bindrcpp       0.2        2017-06-17 CRAN (R 3.3.1)                           
 brew           1.0-6      2011-04-13 CRAN (R 3.3.1)                           
 checkmate    * 1.8.3      2017-07-03 cran (@1.8.3)                            
 colorspace     1.3-2      2016-12-14 cran (@1.3-2)                            
 commonmark     1.2        2017-03-01 CRAN (R 3.3.1)                           
 crayon         1.3.2      2016-06-28 CRAN (R 3.3.1)                           
 data.table   * 1.10.4     2017-02-01 CRAN (R 3.3.1)                           
 datasets     * 3.3.1      2016-09-30 local                                    
 devtools     * 1.13.2     2017-06-02 CRAN (R 3.3.1)                           
 digest         0.6.12     2017-01-27 CRAN (R 3.3.1)                           
 dplyr          0.7.2      2017-07-20 CRAN (R 3.3.1)                           
 ggplot2      * 2.2.1.9000 2017-08-15 Github (hadley/ggplot2@41f154f)          
 glue           1.1.1      2017-06-21 CRAN (R 3.3.1)                           
 graphics     * 3.3.1      2016-09-30 local                                    
 grDevices    * 3.3.1      2016-09-30 local                                    
 grid           3.3.1      2016-09-30 local                                    
 gtable         0.2.0      2016-02-26 cran (@0.2.0)                            
 htmltools      0.3.6      2017-04-28 CRAN (R 3.3.1)                           
 htmlwidgets    0.9        2017-07-10 cran (@0.9)                              
 httr           1.3.1      2017-08-20 cran (@1.3.1)                            
 imbsaddins   * 0.1.0      2017-05-05 local                                    
 jsonlite       1.5        2017-06-01 CRAN (R 3.3.1)                           
 knitr        * 1.17       2017-08-10 cran (@1.17)                             
 lattice        0.20-35    2017-03-25 CRAN (R 3.3.1)                           
 lazyeval       0.2.0      2016-06-12 cran (@0.2.0)                            
 lhs            0.14       2016-08-09 CRAN (R 3.3.1)                           
 magrittr       1.5        2014-11-22 CRAN (R 3.3.1)                           
 Matrix         1.2-10     2017-04-28 CRAN (R 3.3.1)                           
 mco            1.0-15.1   2014-11-29 CRAN (R 3.3.1)                           
 memoise        1.1.0      2017-04-21 cran (@1.1.0)                            
 methods      * 3.3.1      2016-09-30 local                                    
 misc3d         0.8-4      2013-01-25 CRAN (R 3.3.1)                           
 mlr          * 2.12       2017-08-23 Github (mlr-org/mlr@e23a8cb)             
 mlrMBO       * 1.1.0      2017-05-12 CRAN (R 3.3.1)                           
 munsell        0.4.3      2016-02-13 cran (@0.4.3)                            
 parallel       3.3.1      2016-09-30 local                                    
 parallelMap    1.4        2017-07-25 local                                    
 ParamHelpers * 1.11       2017-08-22 Github (berndbischl/ParamHelpers@4f5384d)
 pkgconfig      2.0.1      2017-03-21 CRAN (R 3.3.1)                           
 plot3D         1.1        2016-01-13 CRAN (R 3.3.1)                           
 plotly         4.7.0      2017-05-28 CRAN (R 3.3.1)                           
 plyr           1.8.4      2016-06-08 cran (@1.8.4)                            
 prettyunits    1.0.2      2015-07-13 CRAN (R 3.3.1)                           
 progress       1.1.2      2016-12-14 CRAN (R 3.3.1)                           
 purrr          0.2.2.2    2017-05-11 CRAN (R 3.3.1)                           
 R6             2.2.2      2017-06-17 CRAN (R 3.3.1)                           
 ranger       * 0.8.0      2017-06-20 CRAN (R 3.3.1)                           
 rappdirs       0.3.1      2016-03-28 CRAN (R 3.3.1)                           
 RColorBrewer   1.1-2      2014-12-07 cran (@1.1-2)                            
 Rcpp           0.12.12    2017-07-15 cran (@0.12.12)                          
 reshape2     * 1.4.2      2016-10-22 cran (@1.4.2)                            
 RJSONIO        1.3-0      2014-07-28 CRAN (R 3.3.1)                           
 rlang          0.1.1      2017-05-18 cran (@0.1.1)                            
 roxygen2     * 6.0.1      2017-02-06 CRAN (R 3.3.1)                           
 scales         0.4.1.9002 2017-08-14 Github (hadley/scales@9834874)           
 smoof        * 1.5        2017-04-26 CRAN (R 3.3.1)                           
 splines        3.3.1      2016-09-30 local                                    
 stats        * 3.3.1      2016-09-30 local                                    
 stringi        1.1.5      2017-04-07 CRAN (R 3.3.1)                           
 stringr        1.2.0      2017-02-18 CRAN (R 3.3.1)                           
 survival       2.41-3     2017-04-04 CRAN (R 3.3.1)                           
 testthat     * 1.0.2      2016-04-23 CRAN (R 3.3.1)                           
 tibble         1.3.3      2017-05-28 cran (@1.3.3)                            
 tidyr          0.6.3      2017-05-15 CRAN (R 3.3.1)                           
 tools          3.3.1      2016-09-30 local                                    
 utils        * 3.3.1      2016-09-30 local                                    
 viridisLite    0.2.0      2017-03-24 CRAN (R 3.3.1)                           
 withr          2.0.0      2017-08-15 Github (jimhester/withr@0c4a86d)         
 XML            3.98-1.9   2017-06-19 CRAN (R 3.3.1)                           
 xml2           1.1.1      2017-06-21 Github (hadley/xml2@4be4590)             
 xtable       * 1.8-2      2016-02-05 cran (@1.8-2)

Bug report

ja-thomas commented 7 years ago

Thanks for the example. Could reproduce this.

This happened due to the changes in #1980.

But I'm not sure why the tests didn't catch this prior to merging.

ja-thomas commented 7 years ago

We can close this as PR #1980 was reverted. batchmark should work fine again