Data Library Overhaul

[fvankrieken]

Motivations

Data library is a great starting point for the "extract" portion of dcpy, but there are multiple ways its not meeting our needs. Our main area of focus is data quality, both on the passive/automated side of things as well as reactive. I.e., asking questions like

• Does the latest version of a specific source dataset have schema changes, or other unexpected deviations from the norm?
• Why are we seeing abnormal data in pluto?

Goals

Essentially, we want to go from the current process, roughly this:

To something more like this

With a couple key notes

• We want to archive raw data (#499). Currently, when a script source grabs data and manipulates it, or when we manually load a local csv, we don't keep the actual raw data anywhere. This means we cannot recreate our pipelines from beginning to end if we lose access to some sort of source data. For future flexibility, archiving raw data will be valuable.
• During any transformations/cleanings that happen during data library, we want to keep data in a consistent format in between steps. This will be parquet for now (which can then be loaded into df if pandas is needed for cleaning, or postgis, etc). For this, we need to convert data from raw formats into parquet (#631). This functionality will take the place of what gdal does in data library, but moved upstream before any reprojections, scripting functionality, etc.
• Data at this point should be validated according to our expectations (#650)
• At this point, our transformations can begin.
  1. data cleaning logic in library/scripts #633
  2. additional data cleaning that takes place in "preprocessing" steps of data pipelines #651, and any additional cleaning/preprocessing we want to add
  3. geocoding #634
  4. geospatial - reprojections, choose geometry types, etc #652 2 and 3 are lower priority, 1 and 4 are needed to get an MVP of this finished
• Then, we archive. We might want to include ability to go from parquet back to pg_dump (gdal probably easiest here) or csv. This doesn't get its own issue because it's really just a s3.upload_folder

[damonmcc]

notes from Roadmapping on 2/28

idea for ideal "extract" (aka data-library) process

1. "source" section: archival of raw data
2. "ingestions" of raw data into standard "transformation" format
3. running of processing steps
   • custom per dataset (current library scripts)
   • tabular data cleaning
     • read columns as string, lpad w/ zeroes
     • NAs as empty strings
   • geospatial
     • geocoding via geosupport
     • make_valid
     • set crs
4. export from "transformation" format to any desired formats
5. push to S3

next steps

• could start with archiving raw data
• research alternatives to our uses of GDAL in data-library (file format conversion and reprojection)
• do all this new work in a new folder, not /data-library