mlr-org / mlr

Machine Learning in R
https://mlr.mlr-org.com
Other
1.64k stars 404 forks source link

Duplicate learner ids can be valid #2531

Closed annette987 closed 5 years ago

annette987 commented 5 years ago

I'd like to suggest that duplicate learner ids can sometimes be valid. For example, I am running a benchmark and would like to include lasso and ridge regression, both of which are implemented in glmnet. This is the code I have written:

    library(survival)
    library(mlr)

    vet <- data(veteran)
    mas.task <- makeSurvTask(data = vet, target = c("timeToEvent", "status"))
    ridge.lrn  <- makeLearner(cl="surv.cvglmnet", predict.type="response", alpha = 0, nfolds=10)
    lasso.lrn  <- makeLearner(cl="surv.cvglmnet", predict.type="response", alpha = 1, nfolds=10)
    outer = makeResampleDesc("CV", iters=5, stratify=TRUE)
    learners = list(ridge.lrn, lasso.lrn)
    result = benchmark(learners, mas.task, outer, cindex, show.info = TRUE)

and I get the error message: Error in ensureBenchmarkLearners(learners) : Learners need unique ids!

Here is another example, where I would like to benchmark two different feature selection methods on the same learner: `

cox.lrn <- makeLearner(cl="surv.coxph", predict.type="response")
inner = makeResampleDesc("CV", iters=10, stratify=TRUE)
ctrl.sfs = makeFeatSelControlSequential(method="sfs")
ctrl.ran = makeFeatSelControlRandom(prob=0.5)

cox.wrap.ran.lrn = makeFeatSelWrapper(cox.lrn, resampling = inner, measure = cindex, control = ctrl.ran)
cox.wrap.sfs.lrn = makeFeatSelWrapper(cox.lrn, resampling = inner, measure = cindex, control = ctrl.sfs)

learners2 = list(cox.wrap.ran.lrn, cox.wrap.sfs.lrn)
result = benchmark(learners2, mas.task, outer, cindex, show.info = TRUE)

Again I get the error message: Error in ensureBenchmarkLearners(learners) : Learners need unique ids!

Is this restriction necessary? If so, how can I perform a benchmark on two different versions of the same learner? Or am I doing things the wrong way?

berndbischl commented 5 years ago

hi. this will not be changed. ids are freely user-settable. just select 2 different ids for the learners, and everything works.

nobobdy forces you to select the classname as the id, you can essentailly call the learners "foo" and "bar", or in your case "cox-random", "cox-sfs"