improve pluto archiving

[damonmcc]

• minor pluto releases use dot notation, like 22v3.1
• the bytes URL where they're published uses underscore notation, like 22v3_1
• the data-library template is a bit rigid in that we have to act as if the version is underscore notation to archive it
• this is causing downstream issues and forces a user of the data-library github action to use underscore notation when they never have before/wouldn't expect to

[fvankrieken]

Assuming bytes we mean https://s-media.nyc.gov/agencies/dcp/assets/files/zip/data-tools/bytes/ endpoint

Just to clarify process and where issue occurs

• we publish pluto running from local CLI or github action in db-pluto project

  • this pushes to edm-publish

• at time of data archival, data-library gets data from bytes url to archive data in edm-recipes/dcp-pluto. This is where underscore is in url for file instead of dot notation, meaning the user has to enter the version number with undescore

A couple questions

• is the bytes url just essentially a wrapper for grabbing data from edm-publishing? Could we grab straight from there instead? Or has some sort of transformation occured
• Can we control how bytes is doing this, and not have dot notation messed up for the url?

Don't mean to overcomplicate the issue, just want to make sure that I have the right understanding

[damonmcc]

no worries!

is the bytes url just essentially a wrapper for grabbing data from edm-publishing? Could we grab straight from there instead? Or has some sort of transformation occured

nah the bytes URL isn't a wrapper/redirect for edm-publishing. it points to a server we have that's dedicated to public data access via BYTES of the BIG APPLE. I believe some transformation has occurred and is done by the GIS team but that may be limited to metadata and ESRI related edits.

since all we really seem to archive/use is the primary csv, we may be able to change the source from bytes to edm-publishing. and I'm seeing ~15 other archiving templates that already pull from edm-publishing

Can we control how bytes is doing this, and not have dot notation messed up for the url?

Doubt it lol the WebTeam seems to use underscore notation for all versions in urls