search

mlr-org / mlr

Machine Learning in R
https://mlr.mlr-org.com
Other
1.64k stars 405 forks source link

tune.threshold does not work in makeTuneControlMBO when using multi-point proposals #2184

Closed FlorianPargent closed 6 years ago

FlorianPargent commented 6 years ago

I took this example from the tutorial page of mlrMBO on parallelization and multi-point proposals. The only things I changed is predict.type = "prob" in makeLearner and tune.threshold = TRUE in makeTuneControlMBO.

library(mlr)
library(mlrMBO)
library(parallelMap)

ncpus = 2
iters = 3

par.set = makeParamSet(
  makeNumericParam("cost", -15, 15, trafo = function(x) 2^x),
  makeNumericParam("gamma", -15, 15, trafo = function(x) 2^x)
)

ctrl = makeMBOControl(propose.points = ncpus)
ctrl = setMBOControlTermination(ctrl, iters = iters)
ctrl = setMBOControlInfill(ctrl, crit = crit.ei)
ctrl = setMBOControlMultiPoint(ctrl, method = "cl", cl.lie = min)
tune.ctrl = makeTuneControlMBO(mbo.control = ctrl, 
                               tune.threshold = TRUE)
parallelStartMulticore(cpus = ncpus)
res = tuneParams(makeLearner("classif.svm", 
                             predict.type = "prob"),
                 iris.task, cv3, par.set = par.set, 
                 control = tune.ctrl, show.info = FALSE)
parallelStop()

I get the following error:

Fehler in getOptPathEl.OptPathDF(opt.path, or$best.ind) : Index must be between 1 and 0!

jakob-r commented 6 years ago

Thanks for reporting this bug. Unfortunately I cannot reproduce it using the github versions of mlr and mlrMBO. Which versions are you using?

FlorianPargent commented 6 years ago

I removed all my packages and installed mlr, mlrMBO, and parallelMap from github (DiceKriging, rgenoud, cmaes, e1071 from CRAN). Unfortunately, it is still not working.

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: OS X El Capitan 10.11.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] de_DE.UTF-8/de_DE.UTF-8/de_DE.UTF-8/C/de_DE.UTF-8/de_DE.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rgenoud_5.8-1.0   parallelMap_1.4   mlrMBO_1.1.2      smoof_1.5.1       checkmate_1.8.5   BBmisc_1.11       mlr_2.12          ParamHelpers_1.10

loaded via a namespace (and not attached):
 [1] purrr_0.2.4         DiceKriging_1.5.5   splines_3.4.3       lattice_0.20-35     colorspace_1.3-2    htmltools_0.3.6     viridisLite_0.3.0   survival_2.41-3    
 [9] XML_3.98-1.10       plotly_4.7.1        rlang_0.2.0         e1071_1.6-8         pillar_1.2.0        glue_1.2.0          withr_2.1.1         plot3D_1.1.1       
[17] RColorBrewer_1.1-2  lhs_0.16            mco_1.0-15.1        bindrcpp_0.2        plyr_1.8.4          bindr_0.1           munsell_0.4.3       gtable_0.2.0       
[25] devtools_1.13.5     htmlwidgets_1.0     memoise_1.1.0       misc3d_0.8-4        parallel_3.4.3      curl_3.1            class_7.3-14        Rcpp_0.12.15       
[33] scales_0.5.0        backports_1.1.2     jsonlite_1.5        ggplot2_2.2.1       digest_0.6.15       stringi_1.1.6       dplyr_0.7.4         RJSONIO_1.3-0      
[41] grid_3.4.3          tools_3.4.3         magrittr_1.5        lazyeval_0.2.1      tibble_1.4.2        tidyr_0.8.0         pkgconfig_2.0.1     Matrix_1.2-12      
[49] data.table_1.10.4-3 assertthat_0.2.0    httr_1.3.1          R6_2.2.2            git2r_0.21.0        compiler_3.4.3

I also tried this on my windows computer at home (using parallelStartSocket) with the same result.

javdg commented 6 years ago

Hi, I first encountered this issue on a HPC cluster using a batchtools workflow (with a different learner). I asked @FlorianPargent to reproduce it locally and we managed to narrow it down using the MWE from the tutorial above. It's definitely an issue on different systems and with various configurations. Wild guess: could changes in BBmisc, ParamHelpers or other helper packages be responsible?

FlorianPargent commented 6 years ago

It is still not working for me when including github versions from BBmisc and ParamHelpers.

> devtools::session_info()
Session info --------------------------------------------------------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.4.3 (2017-11-30)
 system   x86_64, darwin15.6.0        
 ui       RStudio (1.1.383)           
 language (EN)                        
 collate  de_DE.UTF-8                 
 tz       Europe/Berlin               
 date     2018-02-28                  

Packages ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 package      * version   date       source                                   
 assertthat     0.2.0     2017-04-11 cran (@0.2.0)                            
 backports      1.1.2     2017-12-13 cran (@1.1.2)                            
 base         * 3.4.3     2017-12-07 local                                    
 BBmisc       * 1.11      2018-02-28 Github (berndbischl/BBmisc@a5a4e45)      
 bindr          0.1       2016-11-13 cran (@0.1)                              
 bindrcpp       0.2       2017-06-17 cran (@0.2)                              
 checkmate    * 1.8.5     2017-10-24 CRAN (R 3.4.2)                           
 class          7.3-14    2015-08-30 CRAN (R 3.4.3)                           
 colorspace     1.3-2     2016-12-14 cran (@1.3-2)                            
 compiler       3.4.3     2017-12-07 local                                    
 data.table     1.10.4-3  2017-10-27 cran (@1.10.4-)                          
 datasets     * 3.4.3     2017-12-07 local                                    
 devtools       1.13.5    2018-02-18 CRAN (R 3.4.3)                           
 DiceKriging    1.5.5     2015-04-24 CRAN (R 3.4.0)                           
 digest         0.6.15    2018-01-28 CRAN (R 3.4.3)                           
 dplyr          0.7.4     2017-09-28 cran (@0.7.4)                            
 e1071          1.6-8     2017-02-02 CRAN (R 3.4.0)                           
 ggplot2        2.2.1     2016-12-30 cran (@2.2.1)                            
 glue           1.2.0     2017-10-29 cran (@1.2.0)                            
 graphics     * 3.4.3     2017-12-07 local                                    
 grDevices    * 3.4.3     2017-12-07 local                                    
 grid           3.4.3     2017-12-07 local                                    
 gtable         0.2.0     2016-02-26 cran (@0.2.0)                            
 htmltools      0.3.6     2017-04-28 cran (@0.3.6)                            
 htmlwidgets    1.0       2018-01-20 cran (@1.0)                              
 httr           1.3.1     2017-08-20 CRAN (R 3.4.1)                           
 jsonlite       1.5       2017-06-01 CRAN (R 3.4.0)                           
 lattice        0.20-35   2017-03-25 CRAN (R 3.4.3)                           
 lazyeval       0.2.1     2017-10-29 cran (@0.2.1)                            
 lhs            0.16      2018-01-04 cran (@0.16)                             
 magrittr       1.5       2014-11-22 cran (@1.5)                              
 Matrix         1.2-12    2017-11-20 CRAN (R 3.4.3)                           
 mco            1.0-15.1  2014-11-29 cran (@1.0-15.)                          
 memoise        1.1.0     2017-04-21 CRAN (R 3.4.0)                           
 methods      * 3.4.3     2017-12-07 local                                    
 misc3d         0.8-4     2013-01-25 cran (@0.8-4)                            
 mlr          * 2.12      2018-02-28 Github (mlr-org/mlr@9922543)             
 mlrMBO       * 1.1.2     2018-02-28 Github (mlr-org/mlrMBO@48fe133)          
 munsell        0.4.3     2016-02-13 cran (@0.4.3)                            
 parallel       3.4.3     2017-12-07 local                                    
 parallelMap  * 1.4       2018-02-28 Github (berndbischl/parallelMap@bda3a81) 
 ParamHelpers * 1.11      2018-02-28 Github (berndbischl/ParamHelpers@59c649e)
 pillar         1.2.0     2018-02-26 cran (@1.2.0)                            
 pkgconfig      2.0.1     2017-03-21 cran (@2.0.1)                            
 plot3D         1.1.1     2017-08-28 cran (@1.1.1)                            
 plotly         4.7.1     2017-07-29 cran (@4.7.1)                            
 plyr           1.8.4     2016-06-08 cran (@1.8.4)                            
 purrr          0.2.4     2017-10-18 cran (@0.2.4)                            
 R6             2.2.2     2017-06-17 CRAN (R 3.4.0)                           
 RColorBrewer   1.1-2     2014-12-07 cran (@1.1-2)                            
 Rcpp           0.12.15   2018-01-20 cran (@0.12.15)                          
 rgenoud      * 5.8-1.0   2017-10-23 CRAN (R 3.4.2)                           
 RJSONIO        1.3-0     2014-07-28 cran (@1.3-0)                            
 rlang          0.2.0     2018-02-20 cran (@0.2.0)                            
 scales         0.5.0     2017-08-24 cran (@0.5.0)                            
 smoof        * 1.5.1     2017-08-14 CRAN (R 3.4.1)                           
 splines        3.4.3     2017-12-07 local                                    
 stats        * 3.4.3     2017-12-07 local                                    
 stringi        1.1.6     2017-11-17 cran (@1.1.6)                            
 survival       2.41-3    2017-04-04 CRAN (R 3.4.3)                           
 tibble         1.4.2     2018-01-22 cran (@1.4.2)                            
 tidyr          0.8.0     2018-01-29 cran (@0.8.0)                            
 tools          3.4.3     2017-12-07 local                                    
 utils        * 3.4.3     2017-12-07 local                                    
 viridisLite    0.3.0     2018-02-01 cran (@0.3.0)                            
 withr          2.1.1     2017-12-19 CRAN (R 3.4.3)                           
 XML            3.98-1.10 2018-02-19 cran (@3.98-1.)                          
>
jakob-r commented 6 years ago

I think it was my mistake. It's reproducible.

This issue touches two things

  1. Multi-class Threshold Tuning was not implemented for mlrMBO
  2. Parallelization disables the possibility to write into the global optpath from within the function that is paralellized. This might affect irace as well. For mlrMBO we can rely on the opt.path of mlrMBO.
jakob-r commented 6 years ago

An not entirely tested fix is available now. You can install it via devtools::install_github("mlr-org/mlr@fix2184_tuneMBO_parallel_threashold")

FlorianPargent commented 6 years ago

Thanks for working on this. Unfortunately I get a new error:

Error in getThresholdFromExtra(el$extra) : object 'ex' not found

jakob-r commented 6 years ago

Should work now

FlorianPargent commented 6 years ago

If I run the example now I get the warning

In mean.default(getThresholdFromExtra(el$extra)$threshold) : argument is not numeric or logical: returning NA

and the threshold is not returned when I print the tuning result object:

Tune result: Op. pars: cost=16; gamma=0.0587 Threshold: NA mmce.test.mean=0.3600000

jakob-r commented 6 years ago

Dang, i am sorry. Can you please try again. This time I had time to test more carefully.

FlorianPargent commented 6 years ago

It seems to work now. Thank you for your help!