Opportunity to reduce redundant evaluation

[zkurtz]

In a simple example of a mixed space optimization with mostly-default parameters and a deterministic objective, mbo repeatedly evaluates the same point:

library(mlrMBO)
library(data.table)

fun = function(x) {
  if(x$method == "a") return(x$number)
  return(1)
}

space = makeParamSet(
  makeDiscreteParam("method", values = c("a", "b")),
  makeNumericParam("number", lower = 0,upper = 2 * pi, requires = quote(method == "a"))
)

obj = makeSingleObjectiveFunction(
  name = "mixed_example",
  fn = fun,
  par.set = space,
  has.simple.signature = FALSE,
  noisy = FALSE,
  minimize = TRUE
)

run = mbo(obj, control = makeMBOControl(), show.info = FALSE)

## ... many warnings of the form:
#           Warning in generateDesign(control$infill.opt.focussearch.points, ps.local,  :
#             generateDesign could only produce 1 points instead of 1000!

DT = as.data.table(run$opt.path)
head(DT[is.na(number), c("method", "number", "y")])

The result shows that one point got evaluated repeatedly, even though I set noisy = FALSE in the objective. Such repeated evaluation is costly in other settings -- is there any reason to allow it?

[jakob-r]

This is indeed suboptimal (also because mbo never jumps to method = a) but also a very special case. We have the setting ctrl = setMBOControlInfill(ctrl, filter.proposed.points = TRUE) which you can activate but apparently we have not implemented it for discrete parameters. This would generate a random proposal when the proposed point equals an already evaluated point. I think we should consider this for discrete parameters as an easy remedy.

Otherwise you could tweak the surrogate and the infill criterion settings. For a low number of discrete values dummy encoding and kriging often is a good choice.

lrn = makeLearner("regr.km", covtype = "matern3_2", optim.method = "gen", control = list(trace = FALSE), predict.type = "se")
lrn = makeDummyFeaturesWrapper(lrn)
lrn = makeImputeWrapper(lrn, classes = list(numeric = imputeMax(10), factor = imputeConstant("<missing>")))
lrn = makeRemoveConstantFeaturesWrapper(lrn)

Note that this will have problems with your given function because of the constant outcome for method = "b".

[jakob-r]

Working on it in #444