DE<>GIS Dataset Review Process + Internal Publishing Versions

Terms

Build: outputs of a build process, e.g. files from the build of Pluto

Draft: A build that has been promoted to the drafts folder, that is ready for review, and is immutable (ie can't be tinkered with or deleted, except for in rare circumstance). Has a draft-version

Draft Version: a sequential version, composed of a number and a summary. e.g. 1-initial-build, 2-fix-the-bug

Published Draft: a Draft moved to publish folder after review and approval.

Published Draft Version: a sequential number, used in the case of republishing. e.g. 1,2,3. It might make sense to format them as v1, v2, etc.?

Code Changes Required

Modify edm-publishing models to account for the draft-version.
Add DraftVersion functionality to dcpy.utils.versions
Change builds to output to edm-publishing / {product-name} / builds / {build-name} Also modify s3 cleanup scripts to clear out the builds folder, not the drafts folder.
Add GH Action to Promote to Draft. The Action should take in a draft summary (e.g. "fix the bug") and dcpy should infer the next draft number, to create the draft-version. This should also create 'Review Required' GH Issue, and assign to GIS.
Modify the Publish GH Action to either publish latest or take a specific draft number.
There needs to exist a link between what's in Drafts and what's in publish, because those versions might not be identical. E.g. When we publish draft: 3-fix-the-thing -> publish: (v)1 we need to document that pub v1 came from draft 3. [TBD on this one]
Clean out the drafts folders. No moving data migration required, I don't think...

BONUS CODE

Let's start preparing to kill the latest folder. Add an endpoint to qaqc to point to a product latest folder.

NYCPlanning / data-engineering