Model checks

[ben18785]

These are essential for the paper as they are ways to check a model's fit to data.

We already do a bit of this in notebooks but we should do much more. The following are quite simple to implement and would highlight many inadequacies in the statistical model (and possibly the mathematical model too):

• [ ] Standardised residuals versus time plots (looks for changes in error over time)
• [ ] Standardised residuals squared versus time plots (looks for changes in the error width over time - indicates that perhaps ARCH or GARCH errors are necessary)
• [ ] Standardised residuals versus y values plots (looks for obvious misfitting)
• [ ] Squared standardised residuals versus y values plots (looks for heteroscedasticity and perhaps a sign that multiplicative errors may be more appropriate than additive)
• [ ] Actual versus predicted bivariate plot. Simple but can be easier to understand than residual plots for diagnosing model issues
• [ ] Cook's distance to identify those points that are likely influential for fit
• [ ] Residual autocorrelation plots (looks for error autocorrelation, which may be a sign that independent errors are inappropriate)
• [ ] qq or pp plots of theoretical error quantiles versus actual (checks that the error process you are using is appropriate; for example, I suspect many electrochemistry models would do better to employ Student-t errors rather than normal; would be nice to show versus binned y values to help looking for heteroscedasticity). Could also do testing of whether there is statistical difference between estimated error process and true error process
• [ ] Posterior quantiles versus data (a nice visual depiction that your model fits the data). Based on this we should also calculate p values indicating the proportion of data points lying within posterior quantiles. Would be nice to calculate a moving window representing model misfit over the time trace -- would highlight those regions where the model is not well captured by data. Could perhaps show this by shading the time trace

[MichaelClerx]

Hi @ben18785 , sounds great, but what are you suggesting we do exactly? Add methods to pints.plot? Write unit tests? Extend the notebook examples for models?

[martinjrobins]

A lot of these checks seem visual, so might be better adding methods to pints.plot?

[MichaelClerx]

Agreed! We had a brief chat about this yesterday :-)

There's precedence for pints.plot doing simulations (a few should be fine, right?) in the series() plot (aka the posterior predictive distribution)

https://github.com/pints-team/pints/blob/master/pints/plot.py#L407

[ben18785]

Cool, sounds good!

On Thu, Feb 7, 2019 at 10:27 AM Michael Clerx notifications@github.com wrote:

Agreed! We had a brief chat about this yesterday :-)

There's precedence for pints.plot doing simulations (a few should be fine, right?) in the series() plot (aka the posterior predictive distribution)

https://github.com/pints-team/pints/blob/master/pints/plot.py#L407

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pints-team/pints/issues/661#issuecomment-461364644, or mute the thread https://github.com/notifications/unsubscribe-auth/AESFqKsovEjgnfB3cQvESXO6r9yYwtodks5vK_97gaJpZM4aka87 .

[martinjrobins]

so what is the TODO list on this issue? Would each of the checkbox list on the issue description be a new notebook demonstrating the particular check? This seems like a chapter of a textbook, rather than a "small issue" ;)