search

tidymodels / yardstick

Tidy methods for measuring model performance
https://yardstick.tidymodels.org/
Other
369 stars 55 forks source link

Feature request: calibration metrics #150

Open UAB-BST-680 opened 4 years ago

UAB-BST-680 commented 4 years ago

I really like yardstick's API for computing discrimination. Calibration is also an important metric for classification models. Is there any interest in developing some yardstick functions to compute a calibration statistic?

Frank Harrell's Unreliability index seems like a good candidate. The Unreliability index is a chi-square statistic with 2 degrees of freedom testing the hypothesis that calibration slope and intercept are ideal (i.e., the slope is 1, the intercept is 0).

alexpghayes commented 4 years ago

On a related note, I would like to request calibration curves, as in, for example, https://scikit-learn.org/stable/auto_examples/calibration/plot_compare_calibration.html#sphx-glr-auto-examples-calibration-plot-compare-calibration-py

ayueme commented 2 years ago

Really need that toooo! Hope calibration curve can be easily plotted with yardstick! Thank you very much.

Guiquan-27 commented 2 years ago

Yes!! This is also I really want. Thanks!

aminadibi commented 2 years ago

Calibration curves are the go-to metric in clinical prediction models, arguably more important that ROC curves.
See https://link.springer.com/book/10.1007/978-3-030-16399-0