Add regional information to the data

[ld-archer]

Need to add spatial information in the form of the Government Office Region (GOR) to the data, using this variable from Understanding Society.

Steps:

• [x] Add the variable to the dataset and format in format raw
• [x] Pipeline: re-factor rate tables for fertility/mortality in daedalus pipeline to the regional level
• [x] Check and fix any issues between Understanding Society using old (pre-1996ish) LADs and Daedalus using newest LAD boundaries. Not expecting any issues here but still need to check

[RobertClay]

Update RateTables in Daeadalus to take Regionals rather than LADs. Can repurporse nolocation scripts.

[RobertClay]

region variable (gor_dv) is split into two variables for some reason (gor_dv_x, gor_dv_y). This isnt in any of the documentation so who knows what this means.

[RobertClay]

also they seem to be exactly equal to each other..

[ld-archer]

I think there's a GOR variable in both indresp and hhresp so that's probably the issue. Can safely ignore the hhresp I reckon.

[RobertClay]

yeah those suffixes indicate its due to a pandas merge. worth looking into dealing with clashes.

[ld-archer]

We can just remove that var as part of the combine function (and any others that clash in future). I'm happy to do it or tell me if you are so we don't muck it up

[RobertClay]

https://stackoverflow.com/questions/62779180/pandas-merge-unexpectedly-produces-suffixes

[RobertClay]

I've added this in to prefer the indresp variables.

[ld-archer]

Looks great!

[RobertClay]

I hate LAD codes. Between 2019 and daedalus rate tables there are several different regions. Some of this may just be wierd naming errors. I'll need to update the JSONs manually I think.

{'Stevenage', 'St Albans', 'Welwyn Hatfield', 'City of London+Westminster', 'Northumberland UA', 'Cornwall+Isles of Scilly', 'Gateshead', 'East Hertfordshire'} {'E07000104', 'E07000101', 'E07000097', 'E06000048', 'E09000001+E09000033', 'E06000052+E06000053', 'E07000100', 'E08000020'}

Regional data for BHPS was easy though so thats nice.

[RobertClay]

https://github.com/tomalrussell/uklad-changes

some hero made this though

[RobertClay]

Also {'Eilean Siar', 'Dumfries & Galloway', 'Derry City and Strabane', 'Perth & Kinross', 'Armagh City, Banbridge and Craigavon', 'Argyll & Bute', 'Edinburgh, City of', 'Ards and North Down'} {'S12000036', 'S12000035', 'N09000011', 'S12000024', 'S12000013', 'S12000006', 'N09000002', 'N09000005'}

I've added all these to downloaded ONS dictionaries. Its sloppy but cant think of a better way without redoing daedalus rate tables from scratch.

[RobertClay]

done. needs reviewing as theres a good chunk of updates.

[ld-archer]

Closed in PR #13