collect_predictions() and calibrators

[topepo]

(or any post-processor that requires training)

Let's exclude data used for training from any performance assessments and these should also be excluded from the results of collect_predictions(). For example:

library(tidymodels)
library(tailor)
library(bonsai)

set.seed(816)
sim_tr <- sim_regression(1000)
sim_rs <- vfold_cv(sim_tr)

reg_mtr <- metric_set(rmse, rsq, mae)

bst_spec <- boost_tree() %>%
  set_mode("regression") %>%
  set_engine("lightgbm")

bst_wflow <- workflow(outcome ~ ., bst_spec)

cal_post <-
  tailor() %>%
  adjust_numeric_calibration(method = "isotonic")

bst_cal_wflow <-
  bst_wflow %>%
  add_tailor(cal_post, prop = .1)

bst_cal_res <- fit_resamples(bst_cal_wflow,
                             sim_rs,
                             metrics = reg_mtr,
                             control = control_resamples(save_pred = TRUE))

# Should _not_ be 1,000
collect_predictions(bst_cal_res) %>% nrow()
#> [1] 1000

manual_calc <- 
  collect_predictions(bst_cal_res) %>% 
  group_by(id) %>% 
  reg_mtr(outcome, .pred) %>% 
  rename(manual = .estimate) %>% 
  select(-.estimator)

check_results <- 
  collect_metrics(bst_cal_res, summarize = FALSE) %>% 
  full_join(manual_calc, by = c("id", ".metric"))

all.equal(check_results$.estimate, check_results$manual)
#> [1] TRUE

^{Created on 2024-09-18 with reprex v2.1.0}

[topepo]

Closing this... There is no issue; the data for the "calibration set" (aka 🥔 😄 ) is shaved off of the analysis/training sets. The data used for performance evaluation are the same so the results above are correct.

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.