Closed bdnffreud closed 6 years ago
What is the sample size? In finite samples the ensemble isn't mathematically guaranteed to have the best cross-validated risk estimate.
The sample size is 9827. The dataset has a high level of imbalance (outcome= 112).
I assume this is an observed data set (not simulation) so you don't know the true AUC for these predictors?
I would say this isn't surprising and can occasionally happen. It is rare for the cross-validation risk estimate from the ensemble to be worse than an individual predictor, but it can happen.
Yes, this is an observed dataset.
Thank you for your help!
I encountered a surprising result with SuperLearner. With a 5-fold cross validation, and method.AUC selected, the Superlearner ensemble AUC average is not the best:
CV.SuperLearner(Y = Y, X = X, V = 5, family = binomial(), SL.library =SL.library, method = "method.AUC", control = list(saveFitLibrary = TRUE), cvControl = list(stratifyCV = TRUE), parallel = cluster)
Risk is based on: Area under ROC curve (AUC)
All risk estimates are based on V = 5
SL.randomForest1000_All 0.73092 NA 0.64227 0.78205 If I exclude SL.glmnet.ridge from the list of algorithms, then the SuperLearner ensemble has the best AUC:
SL.randomForest1000_All 0.73585 NA 0.70240 0.77573
Is there any explanation for the SuperLearner ensemble not outperforming the top base algorithm (SL.glmnet.ridge)?