In order to simplify our depdency management and to create a better separation of concern in the code-base, we need to build a new airflow base image that will allow us to install our requirements for the DAGs in both Reach and other projects without needing to re-build the entire airflow toolchain each time we need a build, as well as reduce the overall size for the web container image.
The new airflow container image can be set up in a new repo and managed separately from the other repositories. This will allow airflow to be specified as a dependency and makes it less tightly ingrained into the code-base itself.
Overall, this should significantly speed up container builds, reduce issues around dependency versioning and management as well a simplify reasoning about the frontend of Reach as an application itself.
Implementation Details
We can base the new Dockerfil on the apache projects airflow Dockerfile https://hub.docker.com/r/apache/airflow/dockerfile here with some modifications (removing features we don't need, using a newer image than slim-stretch).
Set up a Makefile to build the image and push it to our ECR
Set up a trigger on the master branch for the repo to commit to CodeBuild and build the image
In order to simplify our depdency management and to create a better separation of concern in the code-base, we need to build a new airflow base image that will allow us to install our requirements for the DAGs in both Reach and other projects without needing to re-build the entire airflow toolchain each time we need a build, as well as reduce the overall size for the web container image.
The new airflow container image can be set up in a new repo and managed separately from the other repositories. This will allow airflow to be specified as a dependency and makes it less tightly ingrained into the code-base itself.
Overall, this should significantly speed up container builds, reduce issues around dependency versioning and management as well a simplify reasoning about the frontend of Reach as an application itself.
Implementation Details