aws / amazon-mwaa-docker-images

Apache License 2.0
27 stars 11 forks source link

Enforce a mechanism for pip installations that protects Airflow packages #17

Closed rafidka closed 5 months ago

rafidka commented 10 months ago

Overview

We want to make sure that all pip install commands executed in the various scripts don't accidentally break Airflow dependencies. This can be achieved by:

1) Always using Airflow's constraints file. 2) Installing Airflow (and other fundamental packages) with each pip install. This will be a no-op operation, since Airflow is already installed, but it will result in pip trying to make sure there are no conflicts among the packages.

For example, to install a package abc, we do the following:

pip install --constraints ${CONSTRAINT_FILE} "apache-airflow[celery,statsd]==${AIRFLOW_VERSION}" <... other fundamental packages> abc

Acceptance Criteria

Enforce the above somehow. For example, we could create an alias like "safe-pip-install" that adds the constraints and fundamental packages. Thus, the command above would look like this:

safe-pip-install abc

This has to be defined in a global bashrc file, e.g. /etc/bash.bashrc, so the developer doesn't need to redefine it everywhere.