topepo / caret

caret (Classification And Regression Training) R package that contains misc functions for training and plotting classification and regression models
http://topepo.github.io/caret/index.html
1.61k stars 634 forks source link

Custom summary measure with OOB resampling? #1328

Open jarbet opened 1 year ago

jarbet commented 1 year ago

I am trying to use caret to fit a random forest binary classification model using OOB resampling with a custom summary measure. However, I get the following warning:

library(caret);

### custom performance metric function
# https://stackoverflow.com/a/52697940
mySummary  <- function (data, lev = NULL, model = NULL) {
    a1 <- defaultSummary(data, lev, model) # accuracy, kappa
    b1 <- twoClassSummary(data, lev, model) # area under ROC, sens, spec
    c1 <- prSummary(data, lev, model) # area under PR curve, prec, recall, F-score
    out <- c(a1, b1, c1);
    return(out);
  }

# training options
seed = 123
train.ctrl.rf <- trainControl(
    method = "oob",
    classProbs = TRUE,
    summaryFunction = mySummary
    );

Warning message: Custom summary measures cannot be computed for out-of-bag resampling. This value of summaryFunction will be ignored.

Is there any plan to support custom summary measures with OOB resampling?