search

mlr-org / mlrMBO

Toolbox for Bayesian Optimization and Model-Based Optimization in R
https://mlrmbo.mlr-org.com
Other
187 stars 47 forks source link

"Human in the Loop MBO" fails with multiobjective methods "parego" and "nsga2" #474

Open mb706 opened 4 years ago

mb706 commented 4 years ago

parego fails:

ps = makeParamSet(
  makeNumericParam("q", lower = -1, upper = 2),
  makeIntegerParam("v", lower = -2, upper = 3)
)
des = generateDesign(n = 7, par.set = ps)
des$y_1 = c(1.20, 0.97, 0.91, 3.15, 0.58, 1.12, 0.50)
des$y_2 = c(1.20, 0.97, 0.91, 3.15, 0.58, 1.12, 0.50)
ctrl = makeMBOControl(n.objectives = 2)
ctrl = setMBOControlInfill(ctrl, crit = makeMBOInfillCritEI())
ctrl = setMBOControlMultiObj(ctrl, method = "parego")
opt.state = initSMBO(par.set = ps, design = des, control = ctrl, minimize = c(TRUE, TRUE), noisy = FALSE)
proposePoints(opt.state)
x = data.frame(q = 1.7, v = 1)
updateSMBO(opt.state, x = x, y = c(2.19, 2.19))

gives error

Error in control$infill.crit$fun(points = x, models = getOptStateModels(opt.state)[[1]],  : 
  Assertion on 'control$y.name' failed: Must have length 1.

similarly, mspot fails:

ps = makeParamSet(
  makeNumericParam("q", lower = -1, upper = 2),
  makeIntegerParam("v", lower = -2, upper = 3)
)
des = generateDesign(n = 7, par.set = ps)
des$y_1 = c(1.20, 0.97, 0.91, 3.15, 0.58, 1.12, 0.50)
des$y_2 = c(1.20, 0.97, 0.91, 3.15, 0.58, 1.12, 0.50)
ctrl = makeMBOControl(n.objectives = 2)
ctrl = setMBOControlInfill(ctrl, crit = makeMBOInfillCritEI(), opt = "nsga2")
ctrl = setMBOControlMultiObj(ctrl, method = "mspot")
opt.state = initSMBO(par.set = ps, design = des, control = ctrl, minimize = c(TRUE, TRUE), noisy = FALSE)
proposePoints(opt.state)
x = data.frame(q = 1.7, v = 1)
updateSMBO(opt.state, x = x, y = c(2.19, 2.19))

with error

Error in control$infill.crit$fun(points = x, models = getOptStateModels(opt.state)[[1]],  : 
  Assertion on 'control$y.name' failed: Must have length 1.
jakob-r commented 4 years ago

This error occurs because updateSMBO calculates the crit.vals (acquisition function values) for the given x values to write them in the opt.path. This is not essential, but a nice to have.

For the methods you mentioned (and probably more) it is not trivial to calculate them afterwards. I would suggest that we just not calculate the crit.vals in those cases. Using a tryCatch() or ifelse?

mb706 commented 3 years ago

Why does it not crash in mbo()?