Reliability Diagrams, Calibration Error, and Temperature Scaling

[simonschoe]

Hi there,

great work on the package so far! I was wondering if you are planning to extend the types of calibration diagnostics:

• Reliability Diagrams (as outlined in On Calibration of Modern Neural Networks)
• Expected Calibration Error (same paper, related to / derived from Reliability Diagrams)

EDIT: Now that I think about it, isn't a reliability diagram exactly the same plot (using bars instead of a geom_line())?

The former should be easily implemented, right?

segment_logistic %>% 
  mutate(bin = cut(.pred_good, seq(0, 1, 0.1), labels = F),
         .pred_class = if_else(.pred_good > 0.5, "good", "poor")) %>% 
  summarise(accuracy =mean(if_else(Class == "good", 1L, 0L)), .by = bin) %>% 
  ggplot(aes(x = bin, y = accuracy)) +
  geom_col() +
  geom_abline(intercept = 0, slope = 0.1, lty = "dashed") +
  scale_y_continuous(limits = c(0L, 1L)) +
  theme_minimal()

Moreover, do you plan to integrate actual postprocessing steps in the package as well? I am currently working with temperature scaling (in the context of neural networks). I guess its conceptually related to what you were already outlining in the blog post.

Keep on with the great work!

Best, Simon

[topepo]

A lot of this is in the current GH version. Let us know if you see anything missing.

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.