This PR does a lot, basically everything required for me to have confidence pushing facilities to Socrata. it's probably easiest to review commit by commit.
Moves code around to accommodate our new folder structure for lifecycle and models
Renames Product Metadata to align with our new terminology around products and dataset files
Adds Shapefile validation
Adds rowcount validation between different dataset files (e.g. csvs and shapefiles)
Adds attachment validation (just checking that they exist)
Adds Typer CLI to the very top level of dcpy so that we get nice hierarchical layout (and we can remove the spurious "don't invoke me" commands)
Adds validation unit test with an actual shapefile.
Refactors connectors.edm.package to align to new conventions, including our new consensus around where package files should live, as well as the function of edm connector modules.
Validation
Here's what you'll see, using template_db as an example:
['templatedb_points.shp.zip',
'COLUMN_MISMATCH',
"Invalid column(s) found in source data: {'geometry', 'wkb_geomet', "
"'geometry_e'}."]
['templatedb_points.shp.zip',
'COLUMN_MISMATCH',
"Column(s) missing from source data: {'the_geom'}."]
['templatedb_points.shp.zip',
'INVALID_DATA',
'Column bbl contains 218 invalid record(s), for example: None']
['templatedb_polygons.shp.zip',
'COLUMN_MISMATCH',
"Invalid column(s) found in source data: {'geometry', 'wkb_geomet', "
"'geometry_e'}."]
['templatedb_polygons.shp.zip',
'COLUMN_MISMATCH',
"Column(s) missing from source data: {'the_geom'}."]
['templatedb_polygons.shp.zip',
'INVALID_DATA',
'Column bbl contains 2372 invalid record(s), for example: None']
This PR does a lot, basically everything required for me to have confidence pushing facilities to Socrata. it's probably easiest to review commit by commit.
products
anddataset files
dcpy
so that we get nice hierarchical layout (and we can remove the spurious "don't invoke me" commands)Validation
Here's what you'll see, using template_db as an example: