Scraper investigation and implementation for Covid-19.

[ajtucker]

• [x] Investigate missing scrapers.
• [x] Implement missing scrapers.

2d each Shannon & JJ

• [x] NHS Digital fix #79 , #80
• [x] ONS KeyError #82

[ajtucker]

The following datasets still use dataURL in their info.json files:

• NRS-Deaths-involving-coronavirus-COVID-19-in-Scotland #8
• ONS-Number-of-deaths-in-care-homes-notified-to-the-Care-Quality-Commission-England #89
• ONS-Which-occupations-have-the-highest-potential-exposure-to-the-coronavirus-COVID-19 #16
• SG-Coronavirus-Covid-19-additional-data-about-adult-care-homes-in-Scotland #87
• WG-Attendance-at-local-authority-settings-during-the-coronavirus-COVID-19-pandemic #12
• WG-NHS-activity-and-capacity-during-the-coronavirus-COVID-19-pandemic #11
• WG-Summary-data-about-coronavirus-COVID-19-and-the-response-to-it #1
• WG-Survey-of-public-views-on-the-coronavirus-COVID-19 #7
• WG-Testing-data-for-coronavirus-COVID-19 #28

[RedWalters]

The following datasets still use dataURL in their info.json files:

• [x] NRS-Deaths-involving-coronavirus-COVID-19-in-Scotland #8
• [x] ONS-Number-of-deaths-in-care-homes-notified-to-the-Care-Quality-Commission-England #89
• [x] ONS-Which-occupations-have-the-highest-potential-exposure-to-the-coronavirus-COVID-19 #16
• [x] SG-Coronavirus-Covid-19-additional-data-about-adult-care-homes-in-Scotland #87
• [x] WG-Attendance-at-local-authority-settings-during-the-coronavirus-COVID-19-pandemic #12
• [/ ] WG-NHS-activity-and-capacity-during-the-coronavirus-COVID-19-pandemic #11
• [x] WG-Summary-data-about-coronavirus-COVID-19-and-the-response-to-it #1
• [x] WG-Survey-of-public-views-on-the-coronavirus-COVID-19 #7
• [x] WG-Testing-data-for-coronavirus-COVID-19 #28

[RedWalters]

WG-NHS-activity-and-capacity-during-the-coronavirus-COVID-19-pandemic #11 Currently remaining on seed as the seed points to an older ODS file than the most recent, and the most recent file seems to have some form of corruption which breaks the transform.

[RedWalters]

ONS-Number-of-deaths-in-care-homes-notified-to-the-Care-Quality-Commission-England #89 Scraper works but transform script needs updating (New tab in updated dataset)

[RedWalters]

WG-Testing-data-for-coronavirus-COVID-19 #28 Script added to take newest release, however, it has been commented out as the newest release has an issue stopping the file from being transformed Exception: invalid worksheet name "'file://HBA60/HomeB/stats/ETS1/Releases_Bulletins_Publications/Statistical%20Releases/2017/Exclusions/SDR%20XXX_2016%20English.xlsx'#SHCheck_Table03" Will need investigating

[RedWalters]

WG-Attendance-at-local-authority-settings-during-the-coronavirus-COVID-19-pandemic #12 Script added to pull in the most recent dataURL - transform needs updating

[RedWalters]

SG-Coronavirus-Covid-19-additional-data-about-adult-care-homes-in-Scotland #87 Added script to pull the latest dataURL in case of updated dataset

[ajtucker]

@mikeAdamss to look at whether the landing page makes its way through to the eventual metadata in PMD.

[mikeAdamss]

its not, temp scraper overwrites it. Addressing it here: https://github.com/GSS-Cogs/gss-utils/issues/107