search

Ensemble-Automator / conformal-prediction

Quantify uncertainty in any ML model
3 stars 3 forks source link

improper analysis #2

Closed QasimWani closed 3 years ago

QasimWani commented 3 years ago

Binning is required to truly make sense of predictions from conformal. right now, it's more of a CDF type score. What you want is a confidence interval type of score. So, for confidence = .8, bin results from .8 - .9 and see those values, instead of data[confidence ≥ .8]. This is because for confidence of 0, you have the generalization error, which doesn't make sense.

QasimWani commented 3 years ago

Fixed it. See model plot below for test set of loan prediction. image