combine S3 method for caretList objects

[mariosegal]

Can it be possible to allow to add one model to a caretList after some models have been run?

as I understand it, if I run 3 models and then decide to add a fourth I need to rerun the first 3 as well, which could be time consuming (and if we set the resample indices it is also unnecessary)

I tried running them one at a time on a small dataset, but when i combined them they were not in caretList object, and even after I manually changed the class it still does not work, likely because the caretList has additional elements, but who knows

I would even be happy with a way to do it manually

Great package

[zachmayer]

caretList is a list, so you can just add a model to the end of the list:

model_list_big[['my_custom_model']] <- my_custom_model

You'll need to manually make sure the new model uses the same trainControl object as the old models.

Take a look at the vignette for more info: http://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html

[zachmayer]

If that doesn't work, please re-open this with a reproducible example and I'll fix the bug.

[mariosegal]

Thanks for the reply

I tried it and it works, but only if I train the extra model with train(….)

I had tried calling the extra model with caretList with only one model - but combining the lists did not work for me then

my code is below, lovely based on your vignette, I was too lazy to do a triaging set though - feel free to use it for an example

Thanks again

Mario Segal mario.segal@gmail.com

library(caretEnsemble) library('mlbench') library('pROC') data(Sonar) n=5 set.seed(107)

ctrl1 = trainControl(method = "boot", number = n, savePredictions = T, summaryFunction = twoClassSummary, classProbs = TRUE,verboseIter =T, index=createResample(Sonar$Class, n))

list1 = list(glm=caretModelSpec(method='glm',family='binomial'), rpart=caretModelSpec(method='rpart'))

model_list = caretList(Class~.,data=Sonar,trControl = ctrl1, tuneList = list1,metric='ROC')

add a model

list_extra= train(Class~.,data=Sonar,trControl = ctrl1,method='rf')

model_big = model_list #to preserve it model_big[['rf']] <- list_extra model_big

ens1 <- caretEnsemble(model_list) ens2 <- caretEnsemble(model_big)

summary(ens1) summary(ens2)

On Apr 21, 2015, at 11:21 AM, Zach Mayer notifications@github.com wrote:

caretList is a list, so you can just add a model to the end of the list:

model_list_big[['my_custom_model']] <- my_custom_model You'll need to manually make sure the new model uses the same trainControl object as the old models.

Take a look at the vignette for more info: http://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html http://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html — Reply to this email directly or view it on GitHub https://github.com/zachmayer/caretEnsemble/issues/136#issuecomment-94837431.

[zachmayer]

Yup, that makes sense, you have to train the extra models with train. You bring up an interesting new point, though, which is that we may need a combine method for caretList to concatenate 2 of them together.

Try this:

names(model_list) <- paste0(names(model_list), 1)
names(model_big) <- paste0(names(model_big), 2)
new_model_list <- c(model_list, model_big)
class(new_model_list) <- 'caretList'
summary(caretEnsemble(new_model_list))

I'll look into adding a combine function.

[zachmayer]

I'll add a combine S3 method to my todo list. I'll probably just add .x and .y as suffixes if the model names are shared.

[mariosegal]

c() did not work to combine 2 caretList objects, the lists get combined but the class is now List

Mario Segal mario.segal@gmail.com

On Apr 21, 2015, at 12:05 PM, Zach Mayer notifications@github.com wrote:

Yup, that makes sense, you have to train the extra models with train. You bring up an interesting new point, though, which is that we may need a combine method for caretList to concatenate 2 of them together.

See what happens if you try c(en1, end2)?

— Reply to this email directly or view it on GitHub https://github.com/zachmayer/caretEnsemble/issues/136#issuecomment-94854222.

[zachmayer]

Try this:

names(model_list) <- paste0(names(model_list), 1)
names(model_big) <- paste0(names(model_big), 2)
new_model_list <- c(model_list, model_big)
class(new_model_list) <- 'caretList'
summary(caretEnsemble(new_model_list))

[JasonCEC]

I was looking for the same functionality, so I would be happy to help with this given some guidance; what is left to be done?

impetus: Just finished a hyper-parameter search that took 48 hours; I would hate to have to re-run that just to join the model into an ensemble.

[zachmayer]

@JasonCEC I think you just need to write a c.caretEnsemble S3 method. Something like this should work:

names(model_list) <- paste0(names(model_list), 1)
names(model_big) <- paste0(names(model_big), 2)
new_model_list <- c(model_list, model_big)
class(new_model_list) <- 'caretList'
return(new_model_list)

I haven't tested it, but would be happy to accept a PR.

[JasonCEC]

@zachmayer I hope we're talking about the same thing... I understood this bug/feature as a concatenating method for caretList and train objects - not caretEnsemble objects!

[zachmayer]

That sounds correct to me! Thanks for putting this together!