Extension covering idees in "Model Diagnostics and Forecast Evaluation for Quantiles"

[seabbs]

A new paper from @jbracher and friends: Model Diagnostics and Forecast Evaluation for Quantiles (http://dx.doi.org/10.1146/annurev-statistics-032921-020240)

I found this a little hard to read if I am being honest (i.e. it took me a while) so when implementing any of the ideas will need to provide some nice docs vs just relying on pointing at this work.

It has some really nice ideas in it for evaluating calibration and plotting discrimination and miscalibration that we could think about supporting. I assume the authors would be happy with us doing this (@jbracher)?

• Coverage plots w/ consistency and confidence intervals.
• Murphy diagrams (mean score vs the threshold parameter)
• CORP, consistent, optimally binned, reproducible and pool-adjacent-violators- based plots

I particularly like these plots.

[seabbs]

There is also this point from the discussion which covers many of our use cases:

Of course, stronger methods might become available if structure can be exploited; for example, in sequential time-series forecasts for one step ahead, in addition to the predictive quantiles being calibrated, exceedances ought to be independent (Christoffersen 1998).

[nikosbosse]

nice! Still have to read the paper though. Maybe @jbracher and other friends of the group would even be interested in contributing code?

[seabbs]

The paper has code attached which seems like a pretty good starting point for a conversation!

[seabbs]

@jbracher penny for your thoughts!?