DE<>GIS: COLP Packaging

[alexrichey]

Automate the transformations that GIS applies to COLP, as outlined here

Macro Steps:

• [ ] Pull from publishing folder on Digital Ocean
• [ ] Apply transformations
• [x] Validate Data
• [ ] Output to new bucket

Individual Transformations to call out:

• [ ] (Item 4) "XLSX: Convert csv to xlsx. IMPORTANT:This must be done carefully to avoid corrupting data that must retain leading zeros" (AR note: I assume this will happen automatically when we automate this. But maybe not)
• [x] (Item 6) "Confirm/Review data schema is the same as previous version No missing geographic values (i.e. Borough, CD, etc) No blank values in feature class, should be “null” instead "
• [ ] (Item 8) Generate README pdf from sharepoint. Should this reside in (perhaps) markdown in our github repo?
• [ ] (Item 10) "update GDB and SHP metadata, Import metadata from previous version to GDB, Update citation publication date to last day of month (i.e. 6/30/2022)

[damonmcc]

wondering if MyST markdown is a potential way to generate PDFs from markdown

[damonmcc]

outline of packaging stages from GIS doc

1. Download and extract the full output.zip file to \COLP{YYYY}{ YYYYMMDD}\raw_data
   • the shp and gdb data are contained within their own sub-archives. Leave these as-is until ready to process those files
2. Process each dataset in the relevant subfolder
   • GDB: extract from \raw_data directly to \gdb and rename using the following conventions
     • GDB name colp_YYYYMMDD.gdb (e.g. colp_20230802.gdb)
     • Feature class name colp_YYYYMMDD (e.g. colp_20230802)
   • SHP: extract from \raw_data directly to \shp and rename using the following conventions
     • colp_YYYYMMDD.shp (ex. colp_20230802.shp)
   • CSV: copy from \raw_data directly to \csv and rename using the following conventions
     • colp_YYYYMMDD.csv (ex. colp_20230802.csv)
   • XLSX: convert csv to xlsx
     • ...
3. Update readme file from Sharepoint and send to Matt/Amanda for review
4. Receive confirmation from Matt/Amanda that readme is ready for posting
5. Update gdb and shp metadata
6. Zip the csv, xlsx, gdb, and shp files individually with both readme and metadata pdf
7. Save in M:\GIS\BytesProduction\COLP\<YYYY>\< YYYYMMDD>\web
8. Manually export the gdb to SDE Prod

[damonmcc]

thoughts on product release stages

• this seems to be our order of operations
  • Build -> QA -> Publish -> Package -> Distribute
• in general, it seems ideal to do as much as possible during Build so that the steps are automated and QAed
• things like "put version in filename" seem nice to do after QA because consistent filenames help simplify Build and QA automation. these release-based filenames seem only valuable for users to download files after Distribute
• so what's the difference between Publish and Package?
  • Publish seems to be copying a QAed draft, could be considered a "promotion" of data
  • Package seems to be transforming files after QA so that they can be easily distributed
• should we rename and/or eliminate our concept of Publish? Promote may be a better way to describe what we're trying to do to a build that passes QA