search

tidymodels / yardstick

Tidy methods for measuring model performance
https://yardstick.tidymodels.org/
Other
369 stars 55 forks source link

Stratified(Balanced) Brier Score for Imbalanced datasets #506

Open giorgosm3317 opened 6 months ago

giorgosm3317 commented 6 months ago

Hello,

Forgive me if this request has already been made but I could not find it.

The idea is that the ordinary Brier Score in cases of imbalanced datasets may lead to miscalibration of the minority class as mentioned in this paper. The authors propose a stratified version of Brier score giving equal importance to all the classes. In the case of binary classification that would be the unweighted average of the brier scores of the majority and minority class.

It could also be extended to multiclass problems I believe.

Thanks!

EmilHvitfeldt commented 6 months ago

Hello @giorgosm3317 !

This is not a bad idea!