search

tidymodels / baguette

parsnip Model Functions for Bagging
https://baguette.tidymodels.org
Other
24 stars 8 forks source link

OOB error #33

Open datalorax opened 3 years ago

datalorax commented 3 years ago

Feature

Calculate out of bag error and use it for model performance estimates and hyper parameter tuning.

I did a fair amount of sleuthing on this and it looks like this used to be a feature that could be requested through control_bag(), but was removed in 9fae03c3f1dd409e680cf52e63d6efe8f220d7dd because of something related to C5.0.

I'm wondering if we can get this back for rpart models?

Extending this further, I wonder if it might be possible to use tune_grid() with a bagged model, using the OOB samples as the validation set, to tune hyperparameters. I also thought it might be worth thinking about a new function called something like fit_bagged() that would basically operate exactly like fit_resamples(), but would provide the metrics on the OOB samples.

hardin47 commented 2 years ago

For anyone who finds this and wants a workaround, the randomForest engine (not as easy with ranger) allows you to pull the OOB models from the tidy fit. A full reproducible example is here.

topepo commented 11 months ago

We'll look back into this (it was a while back that I pulled that out) and maybe put it on the upcoming user survey.