We want to make sure that all pip install commands executed in the various scripts don't accidentally break Airflow dependencies. This can be achieved by:
1) Always using Airflow's constraints file.
2) Installing Airflow (and other fundamental packages) with each pip install. This will be a no-op operation, since Airflow is already installed, but it will result in pip trying to make sure there are no conflicts among the packages.
For example, to install a package abc, we do the following:
pip install --constraints ${CONSTRAINT_FILE} "apache-airflow[celery,statsd]==${AIRFLOW_VERSION}" <... other fundamental packages> abc
Acceptance Criteria
Enforce the above somehow. For example, we could create an alias like "safe-pip-install" that adds the constraints and fundamental packages. Thus, the command above would look like this:
safe-pip-install abc
This has to be defined in a global bashrc file, e.g. /etc/bash.bashrc, so the developer doesn't need to redefine it everywhere.
Overview
We want to make sure that all
pip install
commands executed in the various scripts don't accidentally break Airflow dependencies. This can be achieved by:1) Always using Airflow's constraints file. 2) Installing Airflow (and other fundamental packages) with each
pip install
. This will be a no-op operation, since Airflow is already installed, but it will result inpip
trying to make sure there are no conflicts among the packages.For example, to install a package
abc
, we do the following:Acceptance Criteria
Enforce the above somehow. For example, we could create an alias like "safe-pip-install" that adds the constraints and fundamental packages. Thus, the command above would look like this:
This has to be defined in a global bashrc file, e.g.
/etc/bash.bashrc
, so the developer doesn't need to redefine it everywhere.