Closed klenwell closed 2 years ago
Test data comes from UsGovCovid19Extract:
See directions at top of that file:
Programmatically extracts Michigan Kent County COVID-19 data from US government API.
Testing data is available on MI website here:
https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html
The request data constants below may need to be updated from time to time. To retrieve
values, do the following:
1. In Firefox, go here: https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html
2. Click TOTAL TESTING tab in dashboard
3. Open Firefox console and go to Network tab. Click trashcan icon to clear log.
4. In County field, select Kent
5. Look for a POST request to the following URL, size will be ~12kb:
https://wabi-us-gov-iowa-api.analysis.usgovcloudapi.net/public/reports/querydata?synchronous=true
6. Right click the item > Copy > Copy as cURL. This will provide the request header and body
data for the API request.
7. Values for WABI constants below will be found in the --data-raw parameter.
There may be less hacky ways to get that data using CDC datasets.
Ok, I have no idea how to track down datasets on the CDC or HHS sites. So I'm going to continue using the hacky approach.
Went here: https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html
Followed instruction above to get this raw cURL:
curl 'https://wabi-us-gov-iowa-api.analysis.usgovcloudapi.net/public/reports/querydata?synchronous=true'
-X POST
-H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0'
[snipped]
--data-raw $'{"version":"1.0.0","queries":[{"Query":{"Commands":[{"SemanticQueryDataShapeCommand":{"Query":{"Version":2,"From":[{"Name":"t","Entity":"Testing Data","Type":0},{"Name":"c","Entity":"Cases by County","Type":0}],"Select":[{"Aggregation":{"Expression":{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"TestCount"}},"Function":0},"Name":"Sum(Testing Data.TestCount)"},{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"MessageDate"},"Name":"Testing Data.MessageDate"},{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"TestType"},"Name":"Testing Data.TestType"}],"Where":[{"Condition":{"In":{"Expressions":[{"Column":{"Expression":{"SourceRef":{"Source":"c"}},"Property":"COUNTY"}}],"Values":[[{"Literal":{"Value":"\'Kent\'"}}]]}}}]},"Binding":{"Primary":{"Groupings":[{"Projections":[0,1]}]},"Secondary":{"Groupings":[{"Projections":[2]}]},"DataReduction":{"DataVolume":4,"Primary":{"Sample":{}},"Secondary":{"Top":{}}},"Version":1},"ExecutionMetricsKind":1}}]},"QueryId":"","ApplicationContext":{"DatasetId":"68c49ce8-850a-41fb-8211-dcf5bd8a7281","Sources":[{"ReportId":"c6f0a0f7-5ac0-4279-bcc2-e47e9439674e","VisualId":"3bfd1db1633430448f1c"}]}}],"cancelQueries":[],"modelId":348880}'
New values?
"DatasetId" : "68c49ce8-850a-41fb-8211-dcf5bd8a7281"
"ReportId" : "c6f0a0f7-5ac0-4279-bcc2-e47e9439674e"
"modelId" : 348880
Problem was header gibberish needed to fetch data had changed and was no longer compatible with what I had. Fixed the issue with some copy paste magic.
I ended up just copying complete --data-raw
string above to this line:
I also updated request headers using values copied from browser console:
8/29 was the last date that included test data. See New Tests column here for example: