Validation

[ld-archer]

First

• [x] Estimate transitions on half the people in ELSA, simulate the other half 2002 - 2016. Compare (T-tests) and evaluate.

Second

• [x] Estimate transitions 2002 - 2008, simulate 2010 - 2016

Third Identify 5 different groups of people Estimate transitions for 4 groups and simulate Redo until everyone has been simulated Ask Bryan for more info on this one

[ld-archer]

Need to sort out naming conventions to make this easier to think about. Going to rename everything related to the current form of cross-validation as CV1, then name the next forms as CV2, CV3 etc. Then we can plan the script(s) needed for CV2 and figure out how to include this in the current workflow. The main aim is to run all forms of cross-validation inside the current scripts, and to replace only what needs to be replaced for that specific version. I think as part of this we need to refactor CV1 work, as there are separate scripts for generating the stock population and imputing in kludge.do.

Steps

• [x] Refactor everything to do with current cross-validation as CV1
• [x] Remove the need for separate CV1 scripts (gen_stock_CV.do, others) and add to the current scripts in workflow
• [x] Plan what is required to carry out CV2, specifically the difference between CV2 and CV1, and how to implement this in workflow
• [x] Add another scenario to ELSA_cross-validation.csv for CV2
• [x] New scenario runs without error (Milestone)
• [ ] Add to T-tests script? Maybe write a new one for this, need to compare ELSA 2010-2016 with FEM 2010-2016

[ld-archer]

Update

Both CV1 and CV2 working as expected, now need to triage and improve based on outputs.

• [x] Run minimal models (but only using gender and age as predictors) (Even more minimal than current)
• [x] Run T-tests script, comparing output from minimal models to ELSA data
  • [x] This will mean slight changes to T-test script to run using different data, use a suffix in Makefile
  • Potential for an R-script for comparing tables? Maybe a nice visualisation
    • [ ] Find literature where T-tests are presented and get some ideas
• [ ] Focus on one model (diabe?) and iteratively include predictors to the minimal model and see where/if simulated output begins to deviate from ELSA
• BMI is a bigger problem, needs a separate issue

[ld-archer]

CV2 Specific Todo The plots we have produced show that age has significant effect on the outputs, and we are missing a replenishing cohort when comparing to ELSA data.

3 ways we could fix this:

• Run a Cohort simulation (not pop.) and focus on prevalence
• Run a Cohort sim, restrict outputs with a minimum age (58? 60?) (Should this be a population sim?)
• Run a population sim, this time including a replenishing cohort to maintain age distribution

This block is now reference in a new issue: #50