Adjust FRS weights to match administrative statistics

[nikhilwoodruff]

The FRS has a well-documented problem with benefit reporting and high incomes: all benefits are under-reported, as well as income sources such as dividends. Up to now, we've operated under the assumption that this is due to measurement error: that recipients are giving incorrect information about their income. However, over time I think we've discovered enough evidence to suggest it is actually mostly sampling error:

• When we calibrate takeup rates to match administrative totals, we are unable to match both caseload and expenditure statistics at the same time, suggesting that the FRS recipients are not highly representative of actual recipients. See #495.
• The FRS questionnaire is highly detailed, with multiple checks on users' answers, and surveyors ask for documented proof of incomes. This still leaves room for error if individuals are not compliant or not aware of benefit receipt, but I doubt to the extent that the data under-reports reality (around 30%).
• UKMOD, the only other microsimulation model that publishes validation statistics, also underestimates benefit aggregates and tax revenues. Under-estimation can be caused by either incorrect data or incorrect modelling: UKMOD and OpenFisca-UK have separate implementations of the same programs, suggesting modelling is not the issue.

After some initial trials, I propose an optimisation-based approach at re-weighting, using TensorFlow to optimise a weight adjustment vector in order to minimise statistical error across a range of validation statistics, penalising substantial divergence from initial weights. Essentially, we want a balance between modified weights that move us as close as possible to benefit and tax statistics, while not moving too far from the initial weights.

Initial experimentation

The following graph illustrates the trade-off between weight edits and statistical error:

And here's an example result from one specific benefit and metric. We can get closer or further, depending on the modification penalty:

Process outline

We'll aim to match the following targets:

• Caseloads and aggregates for all eight major means-tested benefits, plus Income Tax and National Insurance.
• UK and regional household populations

cc @MaxGhenis

[MaxGhenis]

Some additional administrative totals to consider:

• Country-level Income Tax and NICs (also has things like SDLT)
• Taxpayers by band
• Taxpayers, income, and income tax by income range
• Population by age and gender
• Various DWP totals via Stat-Xplore, including their API

[nikhilwoodruff]

Thanks, got all but the last two. I can't seem to find a source for the 5-year age bins in years other than 2020, and I am wondering whether we should include them given they don't uprate directly upwards but mostly along in future years.