ld-archer
commented
3 years ago Update Removed the perturbation, and the outputs were as expected. The prediction was good, and the T-tests p-value was very high, but the model was almost entirely predicted by lag of logbmi.
regress
logbmi
male .0014929
white .0020979
hsless .0006197
college -.002541
l2age65l -.0004071
l2age6574 -.0003958
l2age75p -.0009692
l2logbmi .9929164
_cons .0477836
| Root Mean Square Error
_rmse .0354048The prediction model improved by adding the perturbation back in (or at least looked more realistic), but now probably has gone a bit too far in the other direction:
regress
logbmi
male .0103008
white -.0374643
hsless .0263987
college -.0376384
l2age65l -.0017118
l2age6574 -.0002671
l2age75p -.0035212
l2logbmi .0247247
_cons 3.39122
| Root Mean Square Error
_rmse .1762296The RMSE term here is high. To match the US logbmi model, we need an RMSE of ~0.08, so we need to tweak the bounds for the rnormal distribution (currently -2, 2). Also just to provide some evidence that this change hasn't made some crazy changes down the line, or introduced any systematic error, here is the summary of BMI from the baseline outputs in 2012 and 2036:
. use "/home/luke/Documents/E_FEM_clean/E_FEM/output/ELSA_Baseline/detailed_output/y2012_rep1.dta"
(ELSA_Baseline 2012)
. codebook bmi
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
bmi exp of log BMI
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
type: numeric (float)
range: [14.232368,63.273563] units: 1.000e-07
unique values: 8,267 missing .: 0/10,260
mean: 28.2937
std. dev: 5.31906
percentiles: 10% 25% 50% 75% 90%
22.3992 24.7061 27.534 30.9743 35.0567
. use "/home/luke/Documents/E_FEM_clean/E_FEM/output/ELSA_Baseline/detailed_output/y2036_rep5.dta"
(ELSA_Baseline 2036)
. codebook bmi
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
bmi exp of log BMI
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
type: numeric (float)
range: [14.141178,54.140778] units: 1.000e-06
unique values: 8,453 missing .: 0/8,470
mean: 27.9593
std. dev: 5.11579
percentiles: 10% 25% 50% 75% 90%
21.7853 24.28 27.4627 31.0424 34.7361
The values don't change much over time, but at present the only predictor (aside from sex and age) is the lag of BMI, so this is expected.
The BMI perturbation step has been left alone for a long time, but Bryan has pointed out a potential reason for the poor prediction.
I removed the l2logbmi term from the logbmi transition model, which seemed to make the prediction much better, and the cross validation provided some evidence to support this. However, Bryan explained that this makes the logbmi variable less 'sticky', meaning that if we tried to intervene on a respondent and change BMI, this value would then have no effect on future values. This could mean that interventions on BMI don't work at all, so this needs to be fixed.
There are a few things to do before trying the complicated things:
If the above steps don't stop the crazy over prediction (or make BMI prediction bad in other ways) then we need to try something more complicated. Bryan spoke about doing a 4 year prediction cycle instead, but that would mean anything relying on BMI would also need to be on a 4 year prediction cycle too, which is not very good.