search

klenwell / covid-19

Python command-line application to collect and analyze COVID-19 data.
1 stars 0 forks source link

Kent County stopped reporting test data. #50

Closed klenwell closed 3 years ago

klenwell commented 3 years ago

8/29 was the last date that included test data. See New Tests column here for example:

klenwell commented 3 years ago

Test data comes from UsGovCovid19Extract:

https://github.com/klenwell/covid-19/blob/0906c7bfbcdf4a1511718d794ad423ca9d06a4a7/covid_app/extracts/us_gov_covid19.py

See directions at top of that file:

Programmatically extracts Michigan Kent County COVID-19 data from US government API.

Testing data is available on MI website here:
https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html

The request data constants below may need to be updated from time to time. To retrieve
values, do the following:

1. In Firefox, go here: https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html
2. Click TOTAL TESTING tab in dashboard
3. Open Firefox console and go to Network tab. Click trashcan icon to clear log.
4. In County field, select Kent
5. Look for a POST request to the following URL, size will be ~12kb:
   https://wabi-us-gov-iowa-api.analysis.usgovcloudapi.net/public/reports/querydata?synchronous=true
6. Right click the item > Copy > Copy as cURL. This will provide the request header and body
   data for the API request.
7. Values for WABI constants below will be found in the --data-raw parameter.

There may be less hacky ways to get that data using CDC datasets.

klenwell commented 3 years ago

Ok, I have no idea how to track down datasets on the CDC or HHS sites. So I'm going to continue using the hacky approach.

Went here: https://www.michigan.gov/coronavirus/0,9753,7-406-98163_98173---,00.html

Followed instruction above to get this raw cURL:

curl 'https://wabi-us-gov-iowa-api.analysis.usgovcloudapi.net/public/reports/querydata?synchronous=true' 
  -X POST 
  -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0' 

  [snipped]

  --data-raw $'{"version":"1.0.0","queries":[{"Query":{"Commands":[{"SemanticQueryDataShapeCommand":{"Query":{"Version":2,"From":[{"Name":"t","Entity":"Testing Data","Type":0},{"Name":"c","Entity":"Cases by County","Type":0}],"Select":[{"Aggregation":{"Expression":{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"TestCount"}},"Function":0},"Name":"Sum(Testing Data.TestCount)"},{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"MessageDate"},"Name":"Testing Data.MessageDate"},{"Column":{"Expression":{"SourceRef":{"Source":"t"}},"Property":"TestType"},"Name":"Testing Data.TestType"}],"Where":[{"Condition":{"In":{"Expressions":[{"Column":{"Expression":{"SourceRef":{"Source":"c"}},"Property":"COUNTY"}}],"Values":[[{"Literal":{"Value":"\'Kent\'"}}]]}}}]},"Binding":{"Primary":{"Groupings":[{"Projections":[0,1]}]},"Secondary":{"Groupings":[{"Projections":[2]}]},"DataReduction":{"DataVolume":4,"Primary":{"Sample":{}},"Secondary":{"Top":{}}},"Version":1},"ExecutionMetricsKind":1}}]},"QueryId":"","ApplicationContext":{"DatasetId":"68c49ce8-850a-41fb-8211-dcf5bd8a7281","Sources":[{"ReportId":"c6f0a0f7-5ac0-4279-bcc2-e47e9439674e","VisualId":"3bfd1db1633430448f1c"}]}}],"cancelQueries":[],"modelId":348880}'

New values?

"DatasetId" : "68c49ce8-850a-41fb-8211-dcf5bd8a7281"
"ReportId" : "c6f0a0f7-5ac0-4279-bcc2-e47e9439674e"
"modelId" : 348880
klenwell commented 3 years ago

Resolved

Problem was header gibberish needed to fetch data had changed and was no longer compatible with what I had. Fixed the issue with some copy paste magic.

I ended up just copying complete --data-raw string above to this line:

I also updated request headers using values copied from browser console: