mlr-org / mlr3hyperband

Successive Halving and Hyperband in the mlr3 ecosystem
https://mlr3hyperband.mlr-org.com
GNU Lesser General Public License v3.0
18 stars 5 forks source link

allow initialization with general Sampler from paradox #4

Closed berndbischl closed 4 years ago

SebGGruber commented 4 years ago

Example (please make sure you are on seb-dev branch):

devtools::load_all()
library(mlr3learners)
set.seed(123)

# define hyperparameter and budget parameter for tuning with hyperband
params = list(
  ParamInt$new("nrounds", lower = 1, upper = 16, tag = "budget"),
  ParamDbl$new("eta",     lower = 0, upper = 1),
  ParamFct$new("booster", levels = c("gbtree", "gblinear", "dart"))
)

inst = TuningInstance$new(
  tsk("iris"),
  lrn("classif.xgboost"),
  rsmp("holdout"),
  msr("classif.ce"),
  ParamSet$new(params),
  term("evals", n_evals = 100000)
)

# create custom sampler:
# - beta distribution with alpha = 2 and beta = 5
# - categorical distribution with custom probabilities
sampler = SamplerJointIndep$new(list(
  Sampler1DRfun$new(params[[2]], function(n) rbeta(n, 2, 5)),
  Sampler1DCateg$new(params[[3]], prob = c(0.2, 0.3, 0.5))
))

tuner = TunerHyperband$new(eta = 2L, sampler = sampler)
tuner$tune(inst)

print(inst$archive())
print(tuner$info)
berndbischl commented 4 years ago

here is an example why you should not close this too early: at leats documentation for the Sampler is missing

berndbischl commented 4 years ago

and: if you document this properly a) I dont need the lengthy example code in the issue here, i think. i don't want to complain too much about it, an example usually never hurts b) consider adding this to "@examples" for other users

berndbischl commented 4 years ago

please add

a) some assert that the sampler exactly contains the same params of the paramset, without the budget.

b) a unit test that assert is there

i dont know i removed the line that the budget param can be in the Sampler and is ignored, in docs. usually i dont like this. not sure whats best here

SebGGruber commented 4 years ago

a) and b) is done

juliambr commented 4 years ago
jakob-r commented 4 years ago

I think it is done