Integrate scrapers from prefecture stopcovid19 teams to automate initial data collection

[l15n]

This is a placeholder. I'll add details as I learn more

Motivation

• Case count is increasing rapidly. We're likely to become overwhelmed if we only have manual data entry
• Several other teams working on prefecture-level covid19 projects have automated scraping and ingestion of the prefecture level reports. If we can integrate these, we don't have to get drowned in figuring out how to scrape every single prefecture

Approach

Just the gist of the approach

• Integrate prefectures individually, starting with the prefectures with the highest daily case counts
• Add as a baseline for data entry?

Known scrapers / data sources

WIP

• MHLW from https://www.stopcovid19.jp/
  • scraping, api: https://github.com/code4sabae/app_sabae_cc
  • all data: https://github.com/code4sabae/covid19/blob/master/data/covid19japan-all.json
  • Bonus: Collection of potential primary sources by prefecture https://docs.google.com/spreadsheets/d/1SPqnO0yLn8ubax96sDJZVDcjAH8QT1suLCIgroPGVHY/edit#gid=0
• Tokyo:
  • Imports directly from Excel. Probably with this: https://github.com/tokyo-metropolitan-gov/covid19/tree/development/tool
  • Direct link to data: https://raw.githubusercontent.com/tokyo-metropolitan-gov/covid19/master/data/data.json
• Saitama:
  • Open Data: https://opendata.pref.saitama.lg.jp/
  • unfortunately, the opendata link is not frequently updated, and PDF seems to be the main location. Yet to be re-automated?
  • PDF Source: https://www.pref.saitama.lg.jp/a0701/covid19/documents/youseisyaichirannzeroyonn.pdf
• Hyogo:
  • blog article: https://qiita.com/y-chan/items/87919fd712d1560c5e95
  • Scraping source: https://github.com/stop-covid19-hyogo/covid19-scraping

[liquidx]

So I have a sort-of-working version of this now in the statusboard.js that mostly scraps the aggregate numbers. So it kind of works if you use the "Populate" button on

https://data.covid19japan.com/statusboard/

There are still about 10 prefectures where some of this data is coming from a PDF or image. That problem I don't have a good solution for.

It's all pretty hacky and needs daily attention to update when the site changes structure, but it definitely helps speed up updating the aggregate numbers we don't get from patient data.