ENH: prediction table based on 3 decision including "inconclusive" for binary and multinomial

[josef-pkt]

I had this idea a long time ago, but didn't find an issue.

Prediction tables for binary models like Logit or Multinomial models like MNLogit, OrderedModel pick the choice with the highest probability. Instead we could include an inconclusive region around prob = 0.5 (in binary case), and compute the prediction table only for observations with max probabilities large enough.

I haven't tried an example yet, and didn't see in the literature (but I never looked at many of the binary choice prediction accuracy measures)

related

1577 classification performance measures

also agreement measures, e.g. https://www.statsmodels.org/dev/stats.html#interrater-reliability-and-agreement

I'm currently looking at some residual definitions and gof for Ordinal and similar models. specifically #7076 OrderedModel currently has a method for the standard argmax prediction table.

maybe: Some small classes/categories might have no predicted observations. Maybe looking at second choices might identify cases that are "almost" choosing one of those categories. In multinomial models it is more difficult to come up with asymmetric thresholds for a 0-1 choice or classification.

[josef-pkt]

related: It might be interesting to look at concentration indices, like Herfindahl or Gini, for the multinomial or ordinal predictive distribution. (Distance from the center in a simplex or compositional data ?) I'm not familiar with that literature, but it might give a measure about how "precise" a classification prediction would be.

The interrater and agreement measures might also work directly for multinomial and ordinal distributions, instead of classifications. Those correspond to some multinomial or ordinal measures of association and correlation. I never looked at that for a distribution case.