Closed EricTurner3 closed 3 years ago
Successfully loaded data into DoltHub!
I updated my settings to Insert NewLines and will do PEP8 formatting and unit tests in the next PR. I just do not want to increase this PR as it functionally is working fine and our pipeline does not currently have specific formatting requirements.
ETL Package refactored to read from
schema.json
file and will now:agency_id
and grab the recorddataset_id
, insert if they do not existdata_type
, verify it exists, then grab the same named table inpdap/data-intake
mapping
inschema.json
to match the columns for thepdap/data-intake
tableschema.json
fileStill work-in-progress:
mapping
from the database to actually enumerate the scraped data records and ingest them intopdap/data-intake