The data dictionary does not match the column names in shapefile [Bug]

[MStuhlmacher]

Describe the problem The column headings defined in the data dictionary on this page (https://chichives.com/data) do not match the column headings in the shapefile that can be downloaded from the same page. For example, the data dictionary says that the tree crown density heading is called "treeCrDn" but in the shapefile this heading is "tree_crow" or maybe "tree_den".

Expected behavior Expected behavior is that the data dictionary would match the headings in the shapefile and that the data dictionary would include all columns in the shapefile.

Environment info:

• Device: Laptop
• Operating System: Windows
• Browser: Firefox

[mradamcox]

Thanks for this ticket @MStuhlmacher. Looked into it further and we needed to update our script that generates the downloadable dataset. Should have the fix for this deployed tomorrow.

[mradamcox]

@bodom0015 the problem goes a little deeper than just the need for auto-generating the public download file. Ultimately, the data dictionary that is shown in the Data page comes directly from the Columns sheet in the main Data Dictionary and Variables file, but these column names may not actually exist in the data (and nothing breaks if they don't). While each entry in the Variables sheet does have a column name that I thought had to link to the Columns sheet, but actually links to the column names in the data sheets themselves. For example, trees_crown_density (shortened to trees_crow during shapefile export) is the name in the data sheet, treeCrDn is what is listed in the Columns sheet.

So! The following things need to happen to bring this all together:

1. Audit all column names in the Columns sheet, and where needed change them to:
   • camelCase
   • <= 10 characters long
2. Update source data sheets to use these new names
   • This will require finding which source data sheet has each column in it, and then actually altering the header.
   • Very importantly, any metadata documents that explicitly mention column names will have to be updated as well
3. Update column name as needed for each entry in the Variables sheet
   • I've added validation that pulls from the list of column names in the Columns sheet, which is why those names need to be set to match the actual data before doing this final step.

After all of this is done, checkout the branch from the linked PR above, and run yarn build:data, then check the following:

• The data headers in the downloadable CSV from the Data page match the Data Dictionary on that page
• All variables are still operational on the map
  • This will require clicking through and trying to view each variable.

Anything else? Ultimately this will greatly improve the coherence of the backend data content within the app.

[mradamcox]

Also please fix the typo in this column name asthemaAdj (should be asthmaAdj).

[mradamcox]

This is now handled, it required a bit of work behind the scenes. The solution I outlined above didn't end up being feasible (we aren't able to change the column headers in the original source data), so here is the solution I came up with:

• The Column field in the Columns sheet now must match exactly the original header name in the source data sheet.
• The Column field in the Variables sheet must now match one of the Column entries in the Columns sheet (added a data validation dropdown to make sure of this)
• The Columns sheet now has an extra field: Export Name, which is where we will define the name of this field in exported data, shapefile, CSV etc.
• The presence of an Export Name value determines whether a variable will be included in the public download file
  • We had added a Downloadable field at one point but that is ultimately redundant so it's gone now.