tidymodels / probably

Tools for post-processing class probability estimates
https://probably.tidymodels.org/
Other
115 stars 15 forks source link

better message for estimation functions when the levels are incomplete or incorrect #103

Closed topepo closed 1 year ago

topepo commented 1 year ago

Closes #94

Different error messages:

library(tidymodels)
library(probably)
#> 
#> Attaching package: 'probably'
#> The following objects are masked from 'package:base':
#> 
#>     as.factor, as.ordered

tidymodels_prefer()
theme_set(theme_bw())
options(pillar.advice = FALSE, pillar.min_title_chars = Inf)

# Previously, these both gave errors of '! Multiclass not supported...yet'

# The estimate function didn't specify all prediction columns: 
segment_logistic %>% cal_estimate_logistic(truth = Class, estimate = .pred_poor)
#> Error in `check_level_consistency()`:
#> ! We can't connect the specified prediction columns to some factor levels (good). The selected columns were .pred_poor. Are there more columns to add in the funciton call?
#> Backtrace:
#>     ▆
#>  1. ├─segment_logistic %>% ...
#>  2. ├─probably::cal_estimate_logistic(., truth = Class, estimate = .pred_poor)
#>  3. └─probably:::cal_estimate_logistic.data.frame(...)
#>  4.   └─probably:::cal_logistic_impl(...)
#>  5.     └─probably:::truth_estimate_map(...)
#>  6.       └─probably:::check_level_consistency(truth_levels, est_map)
#>  7.         └─rlang::abort(msg)

# (Incomplete specification of the columns _does_ work with the plot funcitons tho)

# Added better error for logistic with 3+ classes
hpc_cv %>% cal_estimate_logistic(truth = obs, estimate = c(VF:L))
#> Error in `cal_logistic_impl()`:
#> ! The number of outcome factor levels isn't consistent with the calibration method. The levels were: 'VF', 'F', 'M', 'L'
#> Backtrace:
#>     ▆
#>  1. ├─hpc_cv %>% ...
#>  2. ├─probably::cal_estimate_logistic(., truth = obs, estimate = c(VF:L))
#>  3. └─probably:::cal_estimate_logistic.data.frame(., truth = obs, estimate = c(VF:L))
#>  4.   └─probably:::cal_logistic_impl(...)
#>  5.     └─rlang::abort(msg)

Created on 2023-05-05 with reprex v2.0.2

I've also updated the docs to say when we extend binary methods to multiclass.

github-actions[bot] commented 1 year ago

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.