search

zachmayer / caretEnsemble

caret models all the way down :turtle:
http://zachmayer.github.io/caretEnsemble/
Other
226 stars 74 forks source link

Nested cross-validation #117

Closed zachmayer closed 3 months ago

zachmayer commented 9 years ago

It'd be nice to evaluate the ensemble on a separate test set than the one used to make the ensemble.

jknowles commented 9 years ago

:+1: Absolutely agree with this.

rlesca01 commented 9 years ago

I wrote a basic script a couple functions to do this a few weeks ago (rlesca01/caretNestedCV). Its pretty rough at the moment but it does work. Basically it just does a loop over a function which is very similar to caretList and returns a list of a list of models. There is then a function calculates the OOS performance of the individual models and then another function which returns the OOS of the ensembled models. The main.r script should just run as is with the example data. Obviously, if you're doing 10x CV on the outer loop than everything takes much longer.

zachmayer commented 9 years ago

@rlesca01 If you want to incorporate this into caretEnsemble, feel free to send a PR.

Otherwise, Jared or I will give it a shot soonish.

rlesca01 commented 9 years ago

Hi Zach, yeah I'll take a stab at this and send over a PR, mostly I need to add some tests to the code I have.