Tuning DL discrete 'epsilon' hyper-parameter as discrete param gives me an error

[rgmantovani]

Hi,

I'm trying to adjust some DL hyper-parameters, but something is not working right when I try to tune the epsilon one. It is defined as numeric in the ParamSet object. With the first param set (ps1 and epsilon a numericParam) it works. However, if I specify it as a discreteParam I'm getting an error message:

# example with par set 1 - works
library("mlr")

task  = bh.task
lrn   = makeLearner("regr.h2o.deeplearning")
inner = makeResampleDesc(method = "CV", iter = 3)
outer = makeResampleDesc(method = "CV", iter = 2)
meas  = list(rmse, mse)
ctrl  = makeTuneControlRandom(maxit = 2)

# With this par set it works
ps1 = makeParamSet(
  makeDiscreteParam("rho", values = c(0.9, 0.95, 0.99, 0.999)),
  makeNumericParam("epsilon", lower = -10, upper = -4, trafo = function(x) 10^x) 
)

new.lrn1 = makeTuneWrapper(learner = lrn, resampling = inner,
  measure = rmse, par.set = ps1, control = ctrl, show.info = TRUE)

res1 = benchmark(learners = new.lrn1, tasks = list(bh.task), 
  resamplings = outer, measures = meas, show.info = TRUE)

print(res1)

# Example with par set 2 - does now work

ps2 = makeParamSet(
  makeDiscreteParam("rho", values = c(0.9, 0.95, 0.99, 0.999)),
  makeDiscreteParam("epsilon", values = 10^seq(from=-10, to=-4, by=2))
)

new.lrn2 = makeTuneWrapper(learner = lrn, resampling = inner,
  measure = rmse, par.set = ps2, control = ctrl, show.info = TRUE)

res2 = benchmark(learners = new.lrn2, tasks = list(bh.task), resamplings = outer, 
  measures = meas, show.info = TRUE)

print(res2)

The error message:

Error in `colnames<-`(`*tmp*`, value = colnames(env$path)) : 
  'names' attribute [4] must be the same length as the vector [3]
In addition: Warning messages:
1: In paramValueToString.ParamSet(par.set, x, show.missing.values = !remove.nas) :
  number of items to replace is not a multiple of replacement length
2: In paramValueToString.ParamSet(par.set, x, show.missing.values = !remove.nas) :
  number of items to replace is not a multiple of replacement length

Am I doing something wrong? Does anyone can help me how to solve it ?

Follows my sessionInfo():

R version 3.3.0 (2016-05-03)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.6 (El Capitan)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mlr_2.10         ParamHelpers_1.9

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.6      magrittr_1.5     splines_3.3.0    statmod_1.4.24  
 [5] munsell_0.4.3    lattice_0.20-33  xtable_1.8-2     colorspace_1.2-6
 [9] R6_2.1.2         plyr_1.8.4       dplyr_0.5.0      tools_3.3.0     
[13] parallel_3.3.0   grid_3.3.0       checkmate_1.8.1  data.table_1.9.6
[17] gtable_0.2.0     h2o_3.10.0.3     DBI_0.4-1        htmltools_0.3.5 
[21] ggvis_0.4.3      survival_2.39-4  assertthat_0.1   digest_0.6.10   
[25] tibble_1.1       Matrix_1.2-6     shiny_0.13.2     ggplot2_2.1.0   
[29] bitops_1.0-6     RCurl_1.95-4.8   mime_0.5         parallelMap_1.3 
[33] stringi_1.1.1    BBmisc_1.10      scales_0.4.0     backports_1.0.3 
[37] jsonlite_1.0     httpuv_1.3.3     chron_2.3-47  

Thanks.

[berndbischl]

i found the bug. it is in paramhelpers. this is a hard one. we need to discuss this.

[berndbischl]

@rgmantovani thx a lot for finding this and the reproducible examples

1) explanation: the bug is triggered in ParamHelpers. it come from the extremely small values you are using

2) it is fixed in PH on github now. i will push PH to cran tomorrow, then also a new version of mlr. hopefully. you should now be able to repair it by installing PH from GH. but i recommend to wait for the official new release

3) i have added unit tests in mlr AND PH to guard against this.

[rgmantovani]

Thanks @berndbischl