Environment norms/standardization

[fvankrieken]

Another brainstorming issue. Prior to our monorepo, we were generally moving towards dev containers. Do we want one main dev container that we use for all parts of the repo? This also then ties in to environments for github actions - are we pulling from a custom built docker image? Running setup.sh scripts?

Some thoughts 1 vs multiple dev container definitions

Pros of 1 dev container definition/Cons of multiple:

• don't have to build new containers for different projects, once you've built for one you don't need to for others
• eliminate some potential redundancies of having the same things installed in multiple containers, storage concerns
• overall simplicity - keep everything using same versions of things, etc

Cons (or some pros of having multiple container definitions)

• More flexibility per project with multiple dev containers
• Maybe less alignment with cloud environment - for github actions, we probably want to be using minimal overhead for producing datasets, not pulling in some massive image
• Potentially more flexibility if we go with multiple - geosupport has a big image, would be nice to not necessarily use in our "standard" image
• If we have 1 mega-container, it might be slightly tougher to keep track of what's actually needed for each project (easy to work around, write more explicit documentation, but still something to think about
• potentially more annoying to iterate - if it's a long time to pull/build and we need to test new changes, slower cycle

Random thoughts around answers

• Have one general mega-container for local development, have geosupport, latest gdal, etc. It seems as long as we pull from latest gdal, we have all the functionality we need with just a few lines of bash (rather than needing to use a gdal docker image) so wouldn't be hard to just add that on to geosupport docker image.
• publish maybe 3-4 images to dockerhub. These can be used in github actions, as well as if someone needs to pull something lighter weight for their dev container
  • a "base" image - no gdal, no geosupport, but has mc, python (and the packages we use) and other things our most basic pipelines depend on
  • gdal (base + gdal-bin, gdal python)
  • geosupport (current geosupport one maybe fine? Or add on mc)
  • "full" - gdal, geosupport, etc

@damonmcc

[fvankrieken]

And on top of that, a bunch of small important details

• python dependencies across projects
• consistency for things like setup scripts, install scripts, dockerfiles - what goes where
• setting of env variables consistently - we're mostly there, but still something to think about if we change things up

[fvankrieken]

Couple tangential notes on gdal -

• looks like we cannot get 3.6 via apt/apt-get, only 3.4. I found some docs that said to use ppa:ubuntugis/ppa as source to get latest gdal-bin, but seems that's not needed any more, but that regardless of source the latest available is 3.4. So need a slightly more manual install if we're not using a docker image

[fvankrieken]

Series of issues at end of this PR sort of summarize issue

https://github.com/NYCPlanning/data-engineering/pull/15

Attempting to get fgdb working for pluto - could use a base image and install gdal from command line, but more manual/longer process than using docker image. But the published docker images don't come with postgres and other utils, so going in some circles around what things I'm choosing an image with vs what I'm installing vs command line. A little bit of a me issue here, I'll get it settled out and it'll be fine. But still, thinking that we should just publish some images that we like and try to use just those with minimal configuration outside of those image definitions

[damonmcc]

noting that Postgres 14 may be the best unifying version (link)

[fvankrieken]

Random small note that I think makes sense to add here, on the topic of environment and mc setup

Simplest way to handle .env stuff in dev container might just be using envfile option in docker compose. Then devcontainer.json could specify a "setup.sh" script to run at startup (like we've often done) which adds mc spaces using the secrets. I would say we then have the convention that "setup.sh" is something that's supposed to be called both at dev container startup and as a first step of any cloud build step

Side note, in that docker link they mention secrets as being a more appropriate way to store secrets than an envfile/env variables. So maybe worth going that route - just seems a tad more manual (expects secrets to be declared individually, with values being in their own files), so would need to work on ergonomics - maybe generating files from .env as a "pre-build script" or whatever the term is in devcontainer.json

[damonmcc]

noting some thoughts on the various docker images we maintain and where they are located (link)

[damonmcc]

noting that Postgres 14 may be the best unifying version (link)

our persistent DB called EDM-DATA appears to be Postgres 15.3. I checked by running the query select version();. I'm not aware of this causing any issues when changing a build script from using a temporary Postgres 14 DB to using EDM-DATA, but seems good to know

if we eventually want/need to do it, Max recently rolled the version of EDM-DATA forward and back and a few times I remember it not being too bad

[fvankrieken]

This issue is maybe a bit out of date - we could create a new issue for this, but last outstanding issue for near future in my mind is different python environments per recent back and forth on PR, brought up by @alexrichey.

Basically, would be good to start specifying multiple sets of requirements for different environments

• dev ("full" + maybe only one that can get built not from main branch
• build
• streamlit/qa

Anything else? Should also think a bit about pinning versions. Ideally, we pin versions across these environments for consistency of running code, but then subset for installation. Not sure easiest way to do this, or if there's any machinery that we should make around this.

A related note - at this point, are docker images still the way we want to go with this? There's a nice simplicity for use, but just want to check in if anyone has other thoughts

[damonmcc]

@fvankrieken related to the docker maintenance work, is now a good time to create a Dockerhub account using the DataEngineering_DL email? I think we're still using former employees' credentials to push images and such

[fvankrieken]

We are, but it'd have to be associated with the nycplanning org - I've asked @AmandaDoyle for details on that

[fvankrieken]

Also remember that we have a new one - ITD-GDE-DE_DL

[damonmcc]

🎊🎊🎊