Interface for generalizable approach to building prediction intervals

[brshallo]

Would love if tidymodels (in {parsnip} or more likely in a separate pacakge) had support for more generalizable approaches to buiding prediction intervals across choice of model specification (and that, preferably, were less reliant on assumptions compared to parametric approaches).

Reprex for how could look...

library(tidymodels)

### Set-up workflow ###

set.seed(123)
iris <- as_tibble(iris)
split <- initial_split(iris)
train <- training(split)
new_data <- testing(split)

rec <- recipe(Sepal.Length ~ ., data = train)

mod <- parsnip::decision_tree() %>% 
  set_engine("rpart") %>% 
  set_mode("regression")

workflow <- workflows::workflow() %>% 
  add_recipe(rec) %>% 
  add_model(mod) 

### Set-up simulation for predictive inference ###

devtools::source_gist("https://gist.github.com/brshallo/3db2cd25172899f91b196a90d5980690")

# output for a 95% prediction interval
workflow %>% 
  prep_interval(train) %>% 
  predict_interval(new_data, probs = c(0.025, 0.975))

My post on Simulating Prediction Intervals walks through the steps above more explicity. Is just a rough set-up -- could make prep_interval() capable of taking-in tailored resampling structures or other specifications relevant to how the prediction intervals should be generated.

The approaches described in the field of Conformal Inference are relevant (e.g. ryantibs/conformal ). #41 is also tangentially related.

(This issue stems from Rstudio Community thread and Mara's encouragement to open an issue to move discussion to github.)

[brshallo]

Updated post linked to above to use the 0.632+ method for blending the errors in the analysis + assessment sets for the uncertainty due to sample (see gist for bootstrap only implementation).

(Had tried both ways and got similar quality of intervals -- though did not rigorously investigate... 0.632+ method had advantage of not requiring building a separate set of models with cross validation. Moved all the notes and links from the bootstrap + CV based approach to the Appendix.)

[brshallo]

A more simple approach would also be to just use the distribution of the errors on a hold-out set (similar to what {modeltime} currently does).

[brshallo]

Posted funs from gist above into toy package at brshallo/spin (Simulating Prediction INtervals).

[juliasilge]

Thanks for sharing that @brshallo! We're happy that folks are able to extend using tidymodels for methodologies they are interested in. Is it OK if I close this here and you can have discussion related to your work on your repo?

[brshallo]

Yeap, will do.

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.