Update tokens and file structure - WIP

[AmandaDoyle]

Requested OSE changes:

• Change "pct_moe" to "pctmoe"
• Add "_count" to indicate a value is for an estimate, instead of no token
• Add "tpop" to indicate a value is for the total pop and not an specific race/ethnic group, instead of no token
• Organize files such that the year that a column of data corresponds to is consistently communicated by either the file name or by column names.

Level of effort

Change values in the following files:

• aggregate/quality_of_life/diabetes_self_report.py
• aggregate/quality_of_life/access_transit_car.py
• notebooks/household_income_bands_checks.ipynb
• aggregate/quality_of_life/access_to_broadband.py
• aggregate/housing_security/units_affordable.py
• aggregate/PUMS/pums_2000_economics.py
• aggregate/PUMS/pums_2000_demographics.py

Reorganize files to consistently communicate years

Rules:

• If a data point is available for 2 or more time points consistently communicate year.
• If there is a single data point for a single point in time or cumulative across a timespan, then the year is not included.

Option 1: Consistently communicate year in field name

Changes:

• Demographic 9 files become 3 files, with over 600 fields each. Add decennial census year or ACS year token to every field.
• Economic 9 files become 3 files, with nearly 3000 fields each. Add decennial census year or ACS year token to every field.

Option 2: Consistently communicate year in file name

Changes: - QOL and Opportunity Split out health outcome indicators into separate files for each of the following time periods:

• 2000
• 2010
• 2019
• 2000 - 2004
• 2010 - 2014
• 2015 - 2019 Split out public safety indicators into separate files for each of the following time periods:
• 2010 - 2014
• 2016 - 2020 Split out indicators coming from PUMS data for car commute into
• 2008 - 2012
• 2015 - 2019 (add to health)

• Housing security and quality Split out indicators into the following groups
  • 1 time period
  • 2000
  • 2008-2012
  • 2015-2019
  • 2018
  • 2020

[mbh329]

I think the path of least resistance is probably just adding the year to the field name but would that be a cumbersome lift on the data dictionary side?

[AmandaDoyle]

I think the path of least resistance is probably just adding the year to the field name but would that be a cumbersome lift on the data dictionary side?

Noted. I'm not worried about the data dictionary. I am concerned about shoving 3000+ fields into a single file though.

[td928]

I think the path of least resistance is probably just adding the year to the field name but would that be a cumbersome lift on the data dictionary side?

Noted. I'm not worried about the data dictionary. I am concerned about shoving 3000+ fields into a single file though.

For option 1, I agree with Amanda here. We were very hopeful about being able to automate a lot of testing and I think this would certainly add some challenges to that with that front but probably also opportunity -- since manually checking 3000 columns just seem unrealistic now.

Option two on the other hand, would requires new work on exporting files with year, where the year or year range needs to be passed as an additional parameter. As organizationally this seems messy, but it might be overall easier work and easier to manually review the results. But my question is about whether the OSE would accept this format because it does seem quite different from the results from option 1 in my mind.

[SashaWeinstein]

I agree with what's been voiced that avoiding 3k column tables is best if possible