Data library is a great starting point for the "extract" portion of dcpy, but there are multiple ways its not meeting our needs. Our main area of focus is data quality, both on the passive/automated side of things as well as reactive. I.e., asking questions like
Does the latest version of a specific source dataset have schema changes, or other unexpected deviations from the norm?
Why are we seeing abnormal data in pluto?
Goals
Essentially, we want to go from the current process, roughly this:
To something more like this
With a couple key notes
We want to archive raw data (#499). Currently, when a script source grabs data and manipulates it, or when we manually load a local csv, we don't keep the actual raw data anywhere. This means we cannot recreate our pipelines from beginning to end if we lose access to some sort of source data. For future flexibility, archiving raw data will be valuable.
During any transformations/cleanings that happen during data library, we want to keep data in a consistent format in between steps. This will be parquet for now (which can then be loaded into df if pandas is needed for cleaning, or postgis, etc). For this, we need to convert data from raw formats into parquet (#631). This functionality will take the place of what gdal does in data library, but moved upstream before any reprojections, scripting functionality, etc.
Data at this point should be validated according to our expectations (#650)
At this point, our transformations can begin.
data cleaning logic in library/scripts #633
additional data cleaning that takes place in "preprocessing" steps of data pipelines #651, and any additional cleaning/preprocessing we want to add
geocoding #634
geospatial - reprojections, choose geometry types, etc #652
2 and 3 are lower priority, 1 and 4 are needed to get an MVP of this finished
Then, we archive. We might want to include ability to go from parquet back to pg_dump (gdal probably easiest here) or csv. This doesn't get its own issue because it's really just a s3.upload_folder
Motivations
Data library is a great starting point for the "extract" portion of dcpy, but there are multiple ways its not meeting our needs. Our main area of focus is data quality, both on the passive/automated side of things as well as reactive. I.e., asking questions like
Goals
Essentially, we want to go from the current process, roughly this:![image](https://github.com/NYCPlanning/data-engineering/blob/main/docs/diagrams/workflow_data_library.drawio.png?raw=true)
To something more like this![image](https://github.com/NYCPlanning/data-engineering/blob/main/docs/diagrams/desired_workflow_data_library.drawio.png?raw=true)
With a couple key notes
library/scripts
#633