covidatlas / li

Next-generation serverless crawler for COVID-19 data
Apache License 2.0
57 stars 33 forks source link

what are unassigned cases? #357

Closed rtwfroody closed 4 years ago

rtwfroody commented 4 years ago
  {
    "stateID": "iso2:US-ME",
    "tz": "America/New_York",
    "slug": "unassigned-maine-us",
    "population": 1344212,
    "stateName": "Maine",
    "countryName": "United States",
    "name": "Unassigned cases, Maine, US",
    "level": "state",
    "countryID": "iso1:US",
    "locationID": "iso1:us#iso2:us-me#(unassigned)",
jingjtang commented 4 years ago

Are those unassigned cases back-filled? If they are, where could we find the information about the timing that you add the unassigned info? (I noticed that the unassigned numbers don't exist for the most recent date but exist for the previous dates.)

jzohrab commented 4 years ago

Hi @rtwfroody and @jingjtang - sometimes states report cases etc. but don't assign those to a known county.

e.g., for Maine, per the code (src/shared/sources/us/me/index.js), we're scraping a google sheet (https://docs.google.com/spreadsheets/d/13Rbm5zKKLTFNyLZ2Z9YYHc5v6YpO_erMz1pZwiUtfiQ/gviz/tq?tqx=out:csv&sheet=cases_by_county) referenced from the government site https://www.maine.gov/dhhs/mecdc/infectious-disease/epi/airborne/coronavirus/data.shtml. That google sheet contains the following:

"PATIENT_COUNTY","DATA_REFRESH_DT","DATA_AS_OF_DT","CASES","DEATHS","RECOVERIES","HOSPITALIZATIONS"
"Androscoggin","2020-07-30","2020-07-29","547","6","481","40"
...
"Somerset","2020-07-30","2020-07-29","34","1","31","2"
"Unknown","2020-07-30","2020-07-29","1","0","0","0"
"Waldo","2020-07-30","2020-07-29","62","14","47","5"
...

We're mapping "Unknown" to "Unassigned", a common thing that we do throughout our sources.

@jingjtang - re "Are those unassigned cases back-filled?" , we're pulling those from our source data only. We don't back-fill anything. We do occasionally refresh our "timeseries" data, meaning we re-scrape everything, but that's not really backfilling.

I think this answers your questions, so back to you @rtwfroody to close this. If you have more q's, let me know! Cheers, jz

rtwfroody commented 4 years ago

Thanks for the explanation!