search

cmu-delphi / delphi-epidata

An open API for epidemiological data.
https://cmu-delphi.github.io/delphi-epidata/
MIT License
102 stars 68 forks source link

flusurv acquisition is broken #242

Open dfarrow0 opened 4 years ago

dfarrow0 commented 4 years ago

The flusurv data acquisition steps have begun failing with the following stack trace:

current issue: 202040
fetching data for CA (2, 1)
[fetching flusurv data...]
[extracting values...]

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 189, in <module>
    main()
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 185, in main
    update(issue, args.location, args.test)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 99, in update
    data = flusurv.get_data(location_code)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv.py", line 194, in get_data
    data_out = extract_from_object(data_in)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv.py", line 162, in extract_from_object
    prev_rate = data_out[epiweek][age]
IndexError: list index out of range

Tip: reproduce locally by following the dev guide (i.e. bringing up database and web servers) and then running: docker run --rm --network delphi-net delphi_python python3 -m delphi.epidata.acquisition.flusurv.flusurv_update CA

The cause appears to be that the data now, as of 2020w40 (start of the 2020 flu season), includes 3 new age groups. This is great news in general. However, the definition of the new groups is unknown as CDC's webapp doesn't appear to have been updated yet.

Once the site above is updated with definitions of the new age groups, we should capture them. This will require fairly straightforward updates to the acquisition script, the database table, the API server, and the API docs.

In the meantime, as a stop-gap, I'll submit a fix which simply ignores the new, unknown, age groups.

TODO: revisit a week from today -- 2020-09-16

dfarrow0 commented 4 years ago

webapp still not updated, new age groups still undefined. will check back in another week.

nmdefries commented 1 year ago

Age groups 10, 11, and 12 are now defined, along with additional age, sex, and race groups, as:

[
  {'Variable': None, 'valueid': 0, 'parentid': 0, 'Label': 'Overall', 'Color_HexValue': '#000000', 'Enabled': True},

  {'Variable': 'Age', 'valueid': 1, 'parentid': 97, 'Label': '0-4 yr', 'Color_HexValue': '#d19833', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 2, 'parentid': 97, 'Label': '5-17 yr', 'Color_HexValue': '#707070', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 3, 'parentid': 98, 'Label': '18-49 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 4, 'parentid': 98, 'Label': '50-64 yr', 'Color_HexValue': '#516889', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 5, 'parentid': 98, 'Label': '65+ yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 7, 'parentid': 5, 'Label': '65-74 yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 8, 'parentid': 5, 'Label': '75-84 yr', 'Color_HexValue': '#cc5e56', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 9, 'parentid': 5, 'Label': '85+', 'Color_HexValue': '#cc5e56', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 10, 'parentid': 3, 'Label': '18-29 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 11, 'parentid': 3, 'Label': '30-39 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 12, 'parentid': 3, 'Label': '40-49 yr', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 21, 'parentid': 2, 'Label': '5-11  yr', 'Color_HexValue': '#707070', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 22, 'parentid': 2, 'Label': '12-17 yr', 'Color_HexValue': '#707070', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 97, 'parentid': 0, 'Label': '< 18', 'Color_HexValue': '#000000', 'Enabled': True},
  {'Variable': 'Age', 'valueid': 98, 'parentid': 0, 'Label': '>= 18', 'Color_HexValue': '#000000', 'Enabled': True},

  {'Variable': 'Race', 'valueid': 1, 'parentid': None, 'Label': 'White', 'Color_HexValue': '#516889', 'Enabled': True},
  {'Variable': 'Race', 'valueid': 2, 'parentid': None, 'Label': 'Black', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Race', 'valueid': 3, 'parentid': None, 'Label': 'Hispanic/Latino', 'Color_HexValue': '#d19833', 'Enabled': True},
  {'Variable': 'Race', 'valueid': 4, 'parentid': None, 'Label': 'Asian/Pacific Islander', 'Color_HexValue': '#cc5e56', 'Enabled': True},
  {'Variable': 'Race', 'valueid': 5, 'parentid': None, 'Label': 'American Indian/Alaska Native', 'Color_HexValue': '#007d8e', 'Enabled': True},

  {'Variable': 'Sex', 'valueid': 1, 'parentid': None, 'Label': 'Male', 'Color_HexValue': '#44b3c6', 'Enabled': True},
  {'Variable': 'Sex', 'valueid': 2, 'parentid': None, 'Label': 'Female', 'Color_HexValue': '#F2775F', 'Enabled': True}
]
nmdefries commented 1 year ago

Additionally, new strata are available back to ~2009 at least for some locations. We should consider backfilling.