Output folders, dev environments, etc

[fvankrieken]

@alexrichey and I had a bit of a chat today and figured we should start logging thoughts and figure out what we want long term.

Discussion stemmed from questions around logic which currently uses git cli to get branch to use as export folder for build, and how to make this work both in CI as well as local. @alexrichey brought up point that we could move to making these explicit environment variables - in my .env I could have output_folder=fvankrieken or if I really wanted something more fine-grained output_folder=fvankrieken_incorporate_pff etc. But essentially offering more control and decoupling from git slightly (and being able to push to "main" without running code on main, and potentially use inputs in CI for this as well.

@damonmcc curious if you have immediate thoughts

[fvankrieken]

Ties into similar thoughts around schemas/dbs too

[fvankrieken]

Bit of discussion in pff pr here

[damonmcc]

glad to have an issue for this! some thoughts:

• there are a few types of build environments / scenarios that come to mind: local, test in CI, production in CI
• seems like we don't love having to merge code to and build a product on main for it to be QA'd and released. so maybe the usual pattern of {dataset}/{branch}/{version}/... could replace branch with something like the "build environment"
• definitely like being able to define the name of a local build. it's been helpful with schema/DB names
• it seems that no matter the build environment, every build ought to have a version. so we'll probably always need to specify both, with build environment being higher in the chain (e.g. ci_production / 23v1, rather than 23v1 / ci_production

[alexrichey]

I like all these ideas, especially the default of {dataset}/{branch}/{version}/ except with some other concept subbed in for branch. @damonmcc 's suggestion of build_environment makes a lot of sense to me.

Also, happy with the current compromise in which we let folks insert intermediate paths to do something like {dataset}/{build_environment}/ {your/weird/path/here} /{version}/ though maybe we lock that down for certain build_environments like latest or staging.

[alexrichey]

Also, if we extract the path code out of dcpy.edm.publishing.upload, that would be a very self-contained, unit testable bit of code (with some fun edge cases!) we could hand over to @DeaBardhoshi or @athursland

[damonmcc]

noting some links shared in the DE teams channel

we should definitely consider this in out storage/versioning discussions https://docs.digitalocean.com/products/spaces/how-to/versioning/ https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html