Validation

[ld-archer]

Starting this early so we can keep adding to it when we think of things. E.g. add data sources for validation if/when we find them.

[ld-archer]

Household Income

We want to validate household income directly using some external data source, and it might be good to validate the council tax values we generated to use in the calculations. Northern Ireland will be a problem if we validate the council tax values as they calculate their tax differently to the rest of the UK.

The hsctax variable seems to be council tax information including the range for each band. If we see a problem with hh_income then we can look into using this variable and the ranges that go with it.

[RobertClay]

• [x] spatial projection onto chris' data. Particularly for three sipher areas (sheff, gm, scots).
• [ ] limiting probabilities. how does distribution of marginal variable compare with real data over time. expect some shrinkage of variance.
• [ ] time based confints. See Lukes other intervention work. Seaborn/ggplot2 function for it I think.
• [ ] trajectory analysis. Look at longitudinal states of individuals vs reality. Easy for categoricals with TraMiner but continuous more difficult maybe. Spaghetti plots seem simplest.
• [ ] micromacro validation - validation of future data needs external validatoin with other studies. Worth looking into similar studies RE austerity and if we can align experiments to allow for comparison.

[RobertClay]

plan longitudunal validation.

[ld-archer]

Internal Validation

Cross-validation - split dataset in half, use one side to estimate transition models and the other side as a start point for the simulation. See how well our simulated data tracks with real life.

ROC Curve - The only binary absorbing variable we have is death, but this is a very important component to validate. We can use the pROC package in R.

[ld-archer]

External Validation

Comparison With Actual Data

This would be comparing our nowcasting (simulate from 2009-2020ish) to actual data from any source, but best from official outputs and key statistics. Here would be a good place to list any data we find:

Corroboration

Corroborate against other microsimulations - Scottish government microsims have done some similar interventions using UKMOD that we could compare our own outputs to.

[ld-archer]

PRORITY::

Big facet grid of cross-validation plots. Continuous line plots for hh_income and SF12, and stacked bar plots for ordinal pathway variables.