What is the suggested way of integrating error estimation into classification models

[hshany]

For regression models, fitted error models can be used to connect uncertainty scores based on distance-based contributions or ensemble variance to give confidence intervals. For classification models, I see that the predicted probability is calibrated upon test set data. But is there a way to incorporate distance or ensemble variance based uncertainty measure into the confidence score like in regression models?

[davidzqhuang]

Thanks for the note, currently there isn't, the predicted probability is the uncertainty measure for classifications.

We would love community input on uncertainty measures for classification models. As of right now, we haven't decided on a strategy going forward.

[hshany]

How about Deep Ensemble? Practical and model agnostic, also applies to both regression and classification models.