NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
14 stars 0 forks source link

each spatial GFT survey question should have a single flag #814

Open damonmcc opened 2 weeks ago

damonmcc commented 2 weeks ago

problem

Currently, we create a flag column for each dataset and some survey questions.

But GIS needs a single flag column and a single set of source geometry tables (e.g points, lots, buffers) for each spatial survey question.

The GFT tool will only show questions and their flag. Values which contributed to a flag will only been shown in the export table.

The exceptions to this 1-to-1 relationship are

approach

We're using the GFT Data Review excel file to document two things:

These tables (especially the mappings table) can inform any changes DE has to make to the GFT build.

We've had the Data Source review excel file, but that table was used to track source data and doesn't represent the one-to-many relationship of survey question to source datasets

damonmcc commented 2 weeks ago

GIS diagram showing flow from GFT table made by DE to the app and to the two app outputs: Report PDF and Output CSV

image

damonmcc commented 2 weeks ago

will start using the list of survey questions to ensure we're generating a single flag for each one. this is not blocked by confirmation of the source data for each flag