valorumdata / cmdc-tools

3 stars 4 forks source link

Some alaska counties not getting new data. #122

Closed mikelehen closed 4 years ago

mikelehen commented 4 years ago

At least a couple FIPS haven't gotten new data in 7 days (Sep 28): 02016,02070

These seem to have last updated Sep 30: 02195, 02198, 02100 (not exhaustive)

mikelehen commented 4 years ago

It looks like we lost all the real data between 2020-09-28 and 2020-10-04 and instead there's one big discontinuous jump from 2020-09-27 to 2020-09-28. Is this expected and the best we can do? If so, that's probably okay. I just want to confirm.

  {
    "dt": "2020-09-25",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1839,
    "fips": "02016"
  },
  {
    "dt": "2020-09-26",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1840,
    "fips": "02016"
  },
  {
    "dt": "2020-09-27",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1841,
    "fips": "02016"
  },
  {
    "dt": "2020-09-28",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1872,
    "fips": "02016"
  },
  {
    "dt": "2020-09-29",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1872,
    "fips": "02016"
  },
...
  {
    "dt": "2020-10-04",
    "location": 2016,
    "variable": "negative_tests_total",
    "value": 1872,
    "fips": "02016"
  }
mikelehen commented 4 years ago

FWIW- If the discontinuous jump happened on 2020-10-04 (when we started getting data again) instead of on 2020-09-28, it would work better with our pipeline. We have code that deals with "stalled" values by taking the post-stall value and smearing it backwards in time to erase the stall.

i.e. Ideally instead of being stalled at 1872 from 2020-09-28 through 2020-10-04, it would have stalled at the prior value (1841).

sglyon commented 4 years ago

Hey @mikelehen I can't replicate...

❯ http "https://api.covidcountydata.org/covid_us?location=eq.2016&order=dt.desc&limit=50&variable=eq.negative_tests_total" Accept:text/csv | xsv table
dt          location  variable              value  fips
2020-10-05  2016      negative_tests_total  1874   02016
2020-10-04  2016      negative_tests_total  1874   02016
2020-10-03  2016      negative_tests_total  1874   02016
2020-10-02  2016      negative_tests_total  1874   02016
2020-10-01  2016      negative_tests_total  1874   02016
2020-09-30  2016      negative_tests_total  1873   02016
2020-09-29  2016      negative_tests_total  1873   02016
2020-09-28  2016      negative_tests_total  1873   02016
2020-09-27  2016      negative_tests_total  1841   02016
2020-09-26  2016      negative_tests_total  1840   02016
2020-09-25  2016      negative_tests_total  1839   02016
2020-09-24  2016      negative_tests_total  1836   02016
2020-09-23  2016      negative_tests_total  1833   02016
2020-09-22  2016      negative_tests_total  1825   02016
2020-09-21  2016      negative_tests_total  1823   02016
2020-09-20  2016      negative_tests_total  1789   02016
2020-09-19  2016      negative_tests_total  1782   02016
2020-09-18  2016      negative_tests_total  1781   02016
2020-09-17  2016      negative_tests_total  1761   02016
2020-09-16  2016      negative_tests_total  1728   02016
2020-09-15  2016      negative_tests_total  1712   02016
2020-09-14  2016      negative_tests_total  1712   02016
2020-09-13  2016      negative_tests_total  1696   02016
2020-09-12  2016      negative_tests_total  1696   02016
2020-09-11  2016      negative_tests_total  1696   02016
2020-09-10  2016      negative_tests_total  1644   02016
2020-09-09  2016      negative_tests_total  1642   02016
2020-09-08  2016      negative_tests_total  1641   02016
2020-09-07  2016      negative_tests_total  1601   02016
2020-09-06  2016      negative_tests_total  1601   02016
2020-09-05  2016      negative_tests_total  1514   02016
2020-09-04  2016      negative_tests_total  1513   02016
2020-09-03  2016      negative_tests_total  1506   02016
2020-09-02  2016      negative_tests_total  1501   02016
2020-09-01  2016      negative_tests_total  1403   02016
2020-08-31  2016      negative_tests_total  1402   02016
2020-08-30  2016      negative_tests_total  1381   02016
2020-08-29  2016      negative_tests_total  1381   02016
2020-08-28  2016      negative_tests_total  1378   02016
2020-08-27  2016      negative_tests_total  1340   02016
2020-08-26  2016      negative_tests_total  1238   02016
2020-08-25  2016      negative_tests_total  1215   02016
2020-08-24  2016      negative_tests_total  1214   02016
2020-08-23  2016      negative_tests_total  1189   02016
2020-08-22  2016      negative_tests_total  1189   02016
2020-08-21  2016      negative_tests_total  1189   02016
2020-08-20  2016      negative_tests_total  1165   02016
2020-08-19  2016      negative_tests_total  1141   02016
2020-08-18  2016      negative_tests_total  1140   02016
2020-08-17  2016      negative_tests_total  1118   02016
mikelehen commented 4 years ago

FYI- I can't repro either. In fact all of the data I pasted has changed now. So maybe this was fixed upstream.