Closed abossenbroek closed 4 years ago
I updated to the latest git version of mlrmbo (release 1.1.1) and the following code still runs with errors:
require(mlrMBO)
lrn_test = makeFilterWrapper(learner = "regr.xgboost")
param_test = makeParamSet(
makeNumericParam("fw.perc", lower = 0.2, upper = 0.7),
makeDiscreteParam("fw.method", values = "information.gain", tunable = FALSE),
makeDiscreteParam("nrounds", values = 100.0, tunable = FALSE),
makeNumericParam("min_child_weight", lower = 1L, upper = 10L),
makeNumericParam("max_depth", lower = 2, upper = 5, default = 2, trafo = function(x) round(2 * x)),
makeNumericParam("subsample", lower = 0.5, upper = 1),
makeNumericParam("colsample_bytree", lower = 0.4, upper = 1)
)
bh.task_no_factor = createDummyFeatures(bh.task)
mbo.ctrl = makeMBOControl(propose.points = 10,
final.evals = 32)
mbo.ctrl = setMBOControlInfill(control = mbo.ctrl,
crit = cb)
mbo.ctrl = setMBOControlMultiPoint(control = mbo.ctrl, method = "moimbo",
moimbo.objective = "ei.dist",
moimbo.dist = "nearest.neighbor",
moimbo.maxit = 10L)
mbo.ctrl = setMBOControlTermination(mbo.ctrl, iters = 10)
ctrl = mlr:::makeTuneControlMBO(mbo.control = mbo.ctrl)
x = tuneParams(lrn_test, task = bh.task_no_factor, resampling = cv3,
par.set = param_test,
measures = mape,
control = ctrl)
gives the following error,
[Tune] Started tuning learner regr.xgboost.filtered for parameter set:
Type len Def Constr Req Tunable Trafo
fw.perc numeric - - 0.2 to 0.7 - TRUE -
fw.method discrete - - information.gain - FALSE -
nrounds discrete - - 100 - FALSE -
min_child_weight numeric - - 1 to 10 - TRUE -
max_depth numeric - 2 2 to 5 - TRUE Y
subsample numeric - - 0.5 to 1 - TRUE -
colsample_bytree numeric - - 0.4 to 1 - TRUE -
With control class: TuneControlMBO
Imputation value: Inf
[Tune-x] 1: fw.perc=0.571; fw.method=information.gain; nrounds=100; min_child_weight=4.49; max_depth=7; subsample=0.649; colsample_bytree=0.695
[Tune-y] 1: mape.test.mean=0.128; time: 0.0 min
[Tune-x] 2: fw.perc=0.646; fw.method=information.gain; nrounds=100; min_child_weight=3.69; max_depth=7; subsample=0.741; colsample_bytree=0.766
[Tune-y] 2: mape.test.mean=0.129; time: 0.0 min
[Tune-x] 3: fw.perc=0.54; fw.method=information.gain; nrounds=100; min_child_weight=4.97; max_depth=7; subsample=0.885; colsample_bytree=0.821
[Tune-y] 3: mape.test.mean=0.122; time: 0.0 min
[Tune-x] 4: fw.perc=0.415; fw.method=information.gain; nrounds=100; min_child_weight=4.01; max_depth=6; subsample=0.908; colsample_bytree=0.485
[Tune-y] 4: mape.test.mean=0.135; time: 0.0 min
[Tune-x] 5: fw.perc=0.312; fw.method=information.gain; nrounds=100; min_child_weight=8.36; max_depth=6; subsample=0.851; colsample_bytree=0.555
[Tune-y] 5: mape.test.mean=0.131; time: 0.0 min
[Tune-x] 6: fw.perc=0.265; fw.method=information.gain; nrounds=100; min_child_weight=9.4; max_depth=8; subsample=0.572; colsample_bytree=0.98
[Tune-y] 6: mape.test.mean=0.134; time: 0.0 min
[Tune-x] 7: fw.perc=0.297; fw.method=information.gain; nrounds=100; min_child_weight=5.18; max_depth=9; subsample=0.916; colsample_bytree=0.532
[Tune-y] 7: mape.test.mean=0.135; time: 0.1 min
[Tune-x] 8: fw.perc=0.49; fw.method=information.gain; nrounds=100; min_child_weight=7.06; max_depth=8; subsample=0.597; colsample_bytree=0.86
[Tune-y] 8: mape.test.mean=0.13; time: 0.1 min
[Tune-x] 9: fw.perc=0.683; fw.method=information.gain; nrounds=100; min_child_weight=6.6; max_depth=5; subsample=0.764; colsample_bytree=0.897
[Tune-y] 9: mape.test.mean=0.124; time: 0.0 min
[Tune-x] 10: fw.perc=0.46; fw.method=information.gain; nrounds=100; min_child_weight=3.21; max_depth=4; subsample=0.933; colsample_bytree=0.574
[Tune-y] 10: mape.test.mean=0.123; time: 0.0 min
[Tune-x] 11: fw.perc=0.281; fw.method=information.gain; nrounds=100; min_child_weight=2.24; max_depth=6; subsample=0.697; colsample_bytree=0.514
[Tune-y] 11: mape.test.mean=0.137; time: 0.0 min
[Tune-x] 12: fw.perc=0.603; fw.method=information.gain; nrounds=100; min_child_weight=9.19; max_depth=8; subsample=0.663; colsample_bytree=0.647
[Tune-y] 12: mape.test.mean=0.132; time: 0.0 min
[Tune-x] 13: fw.perc=0.508; fw.method=information.gain; nrounds=100; min_child_weight=7.8; max_depth=5; subsample=0.637; colsample_bytree=0.492
[Tune-y] 13: mape.test.mean=0.128; time: 0.0 min
[Tune-x] 14: fw.perc=0.352; fw.method=information.gain; nrounds=100; min_child_weight=1.88; max_depth=4; subsample=0.618; colsample_bytree=0.613
[Tune-y] 14: mape.test.mean=0.127; time: 0.0 min
[Tune-x] 15: fw.perc=0.335; fw.method=information.gain; nrounds=100; min_child_weight=1.13; max_depth=9; subsample=0.715; colsample_bytree=0.83
[Tune-y] 15: mape.test.mean=0.126; time: 0.1 min
[Tune-x] 16: fw.perc=0.654; fw.method=information.gain; nrounds=100; min_child_weight=6.23; max_depth=9; subsample=0.781; colsample_bytree=0.881
[Tune-y] 16: mape.test.mean=0.123; time: 0.1 min
[Tune-x] 17: fw.perc=0.376; fw.method=information.gain; nrounds=100; min_child_weight=2.85; max_depth=8; subsample=0.997; colsample_bytree=0.678
[Tune-y] 17: mape.test.mean=0.128; time: 0.1 min
[Tune-x] 18: fw.perc=0.432; fw.method=information.gain; nrounds=100; min_child_weight=7.46; max_depth=7; subsample=0.965; colsample_bytree=0.436
[Tune-y] 18: mape.test.mean=0.137; time: 0.0 min
[Tune-x] 19: fw.perc=0.389; fw.method=information.gain; nrounds=100; min_child_weight=3.46; max_depth=10; subsample=0.508; colsample_bytree=0.627
[Tune-y] 19: mape.test.mean=0.134; time: 0.1 min
[Tune-x] 20: fw.perc=0.468; fw.method=information.gain; nrounds=100; min_child_weight=4.85; max_depth=6; subsample=0.684; colsample_bytree=0.799
[Tune-y] 20: mape.test.mean=0.13; time: 0.0 min
[Tune-x] 21: fw.perc=0.253; fw.method=information.gain; nrounds=100; min_child_weight=2.35; max_depth=8; subsample=0.871; colsample_bytree=0.933
[Tune-y] 21: mape.test.mean=0.124; time: 0.1 min
[Tune-x] 22: fw.perc=0.622; fw.method=information.gain; nrounds=100; min_child_weight=8.47; max_depth=4; subsample=0.522; colsample_bytree=0.462
[Tune-y] 22: mape.test.mean=0.134; time: 0.0 min
[Tune-x] 23: fw.perc=0.402; fw.method=information.gain; nrounds=100; min_child_weight=7.22; max_depth=5; subsample=0.569; colsample_bytree=0.938
[Tune-y] 23: mape.test.mean=0.128; time: 0.0 min
[Tune-x] 24: fw.perc=0.53; fw.method=information.gain; nrounds=100; min_child_weight=9.71; max_depth=10; subsample=0.787; colsample_bytree=0.962
[Tune-y] 24: mape.test.mean=0.13; time: 0.1 min
[Tune-x] 25: fw.perc=0.674; fw.method=information.gain; nrounds=100; min_child_weight=5.61; max_depth=9; subsample=0.819; colsample_bytree=0.415
[Tune-y] 25: mape.test.mean=0.136; time: 0.1 min
[Tune-x] 26: fw.perc=0.213; fw.method=information.gain; nrounds=100; min_child_weight=8.88; max_depth=5; subsample=0.954; colsample_bytree=0.736
[Tune-y] 26: mape.test.mean=0.138; time: 0.0 min
[Tune-x] 27: fw.perc=0.227; fw.method=information.gain; nrounds=100; min_child_weight=5.95; max_depth=10; subsample=0.547; colsample_bytree=0.761
[Tune-y] 27: mape.test.mean=0.145; time: 0.1 min
[Tune-x] 28: fw.perc=0.583; fw.method=information.gain; nrounds=100; min_child_weight=1.41; max_depth=6; subsample=0.826; colsample_bytree=0.713
[Tune-y] 28: mape.test.mean=0.122; time: 0.0 min
Error in crossover(t(X[parents, , drop = FALSE])) :
Argument 's_parents' is not a real matrix.
In addition: Warning message:
In dist(X) : NAs introduced by coercion
Any advise how to solve this?
makeDiscreteParam("fw.perc", values = 0.6, tunable = FALSE),
why are you setting tunable=FALSE? i guess this is not doing what you think it does. this is an internal markup for the learner. i think you want to fix the param to a value right? then just use setHyperPars for the learner
surrogate.lrn = makeLearner("regr.randomForest", predict.type = "se") surrogate.lrn = makeImputeWrapper(surrogate.lrn, classes = list(numeric = imputeConstant(1e3), factor = imputeConstant("miss")))
why are you not using the mbo default surrogate model here?
I updated to the latest git version of mlrmbo (release 1.1.1) and the following code runs fine,
so your potential problem is resolved? i will still answer the questions that follow now.
why is the learner called 160 times?
10 iters x 10 proposed.points.per.iter = 100
if you want more fine grained control about budget and termination, read setMBOControlTermination
why does the mape change despite the fact that the hyperparameters do not change? I assume the way the cross validation is taken but just want to check.
no. the CV splits are synchronized, and all the same. read ?TuneControl.
same.resampling.instance [logical(1)]
Should the same resampling instance be used for all evaluations to reduce variance? Default is TRUE.
but xgboost is a stochastic algorithm. just consider "col.sampling".......
@berndbischl , great input, thanks for your feedback. I tried to find the right spots in the documentation but sometimes it is a bit hard. The reason why I use a custom surrogate learner is that I also do SVM regression that sometimes yield NA.
@berndbischl : I noted that I had a typo in the code where I thought I solved the issue. Hence I removed my comment, simplified the steps as per your recommendation but still an error. Any advise would be greatly appreciated!
I also ran into this:
obj.fun <- smoof::makeSingleObjectiveFunction(
fn = function(x) checkmate::assertIntegerish(x$ypar),
par.set = mlrCPO::pSS(xpar: integer[0, 10], ypar: integer[0, 10]),
has.simple.signature = FALSE)
ctrl <- makeMBOControl(propose.points = 2)
ctrl <- setMBOControlMultiPoint(ctrl, method = "moimbo")
ctrl <- setMBOControlInfill(ctrl, makeMBOInfillCritEI())
mbo(obj.fun, control = ctrl)
gives the same error:
Computing y column(s) for design. Not provided.
[mbo] 0: xpar=6; ypar=2 : y = 2 : 0.0 secs : initdesign
[mbo] 0: xpar=3; ypar=4 : y = 4 : 0.0 secs : initdesign
[mbo] 0: xpar=7; ypar=10 : y = 10 : 0.0 secs : initdesign
[mbo] 0: xpar=0; ypar=9 : y = 9 : 0.0 secs : initdesign
[mbo] 0: xpar=5; ypar=6 : y = 6 : 0.0 secs : initdesign
[mbo] 0: xpar=8; ypar=0 : y = 0 : 0.0 secs : initdesign
[mbo] 0: xpar=10; ypar=7 : y = 7 : 0.0 secs : initdesign
[mbo] 0: xpar=1; ypar=3 : y = 3 : 0.0 secs : initdesign
Error in crossover(t(X[parents, , drop = FALSE])) :
Argument 's_parents' is not a real matrix.
I try to optimize the run the following optimization
which results in the following output:
Instead of giving the desired outcome. Any suggestions how I could resolve this?
My R session info: