search

lorentzenchr / model-diagnostics

Tools for diagnostics and assessment of (machine learning) models
https://lorentzenchr.github.io/model-diagnostics/
MIT License
28 stars 4 forks source link

Make confidence intervals of reliability diagram tighter #143

Open lorentzenchr opened 8 months ago

lorentzenchr commented 8 months ago

The confidence intervals of plot_reliability_diagram are made a bit too wide by

# We make the interval conservatively monotone increasing by applying
# np.maximum.accumulate etc.
lower = -np.minimum.accumulate(-boot.confidence_interval.low)
upper = np.maximum.accumulate(boot.confidence_interval.high)

This is not conservative because in might lead to the conclusion that the model is auto-calibrated - within the uncertainty - while it actually is not.