Perform posterior predictive checks

[athowes]

• Coverage of posterior predictive intervals i.e. assess what proportion of the observed data falls inside a (1 - alpha)% predictive interval

[athowes]

• This is probably going to be done at the inla.posterior.sample stage
• Could add a column to df like "quantile of the posterior that the data lies in". Then do grouping, summary, plotting etc. of that column

[athowes]

Need now to add coverage tables

[athowes]

• This needs to be corrected (thanks Jeff and Tim): not accounting for additional sampling variability in the estimated probabilities
• Suggest using some kind of likelihood to perform posterior predictive checks
  • Either the Poisson (or xPoisson) on the latent intensity in a univariate manner
  • Or the multinomial (or xmultinomial) on the probabilities

[athowes]

Twitter thread about this: https://twitter.com/adamhowes/status/1455629147217276930

Summary:

• Anderson-Darling-test for uniformity of the PIT-histograms. Test is very conservative and will reject the null hypothesis quite easily
• Could use a proper scoring rule, which would assess calibration and sharpness simultaneously. Issue is that I didn't exactly want to assess them simultaneously. Is there any way to decompose a proper scoring rule into a calibration component and sharpness component? Maybe not:

In terms of scoring rules, they can’t tell you if a prediction is calibrated, just if it’s better than another. (Log scores tend to be a bit more sensitive to the tails than crps iirc. If you are targeting a particular interval, the interval score is great). @dan_p_simpson

• The paper Graphical Test for Discrete Uniformity and its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison. Should try to add the regions onto each of the plots that I have e.g. the following plot

[athowes]

• Potential concern as to the extent to which achieving calibration is about over-fitting the data
• Could consider using out-of-sample

[athowes]

Got the two types of plots in now! Grateful to Säilynoja et al. for their work on this and making code available :)

[athowes]

Happy with these, closing now!