Astronomer Core Docker Images

Astronomer makes it easy to run, monitor, and scale Apache Airflow deployments in our cloud or yours. Source code is made available for the benefit of customers.

Terminology

Note: Edge builds are always development builds

Build matrix

Docker images

Docker images for deploying and running Astronomer Core are currently available on Quay.

We publish 2 variants for each AC Version (example: 2.3.4-7):

1. quay.io/astronomer/ap-airflow:2.3.4-7
2. quay.io/astronomer/ap-airflow:2.3.4-7-onbuild

The only difference between them is that the -onbuild images uses Docker ONBUILD commands to copy packages.txt, requirements.txt and the entire project directory (including dags, plugins folders etc) in the docker file.

We also publish a "floating" or movable tag that points at the latest release of the Airflow version:

1. quay.io/astronomer/ap-airflow:2.3.4
2. quay.io/astronomer/ap-airflow:2.3.4-onbuild

Version Life Cycle & Maintenance policy

The support and maintenance of the Docker images are described in the Version Life Cycle.

Contents of this repo

• The official Dockerfiles that build Astronomer Core Images
• Example docker-compose files for running various pieces and configurations of the platform.

Contribute

• Source Code: https://github.com/astronomer/ap-airflow
• Issue Tracker: https://github.com/astronomer/ap-airflow/issues

Step-by-step instructions for common activities

Release a new Astronomer Certified major, minor, or bugfix version (eg: X.Y.Z)

Release an existing Astronomer Certified version with an updated version of Airflow

Add new Astronomer Certified development version

Add a new base build image (eg: new Debian stable release)

Changelog

All changes applied to available point releases will be documented in the CHANGELOG.md files within each version folder:

• 2.3.4 Changelog
• 2.4.3 Changelog

Testing

Local testing

This testing will run automatically in CI, but it will save some time to try it out locally first.

Airflow is launched into a local Kubernetes cluster using the project "kind" and the most recent version of the Astronomer airflow chart. Python's 'testinfra' module is used to perform system testing on the components while they are running in "kind".

Ensure prerequisites are met:

• docker
• python3
• virtualenv

Ensure docker installed, and user has permissions

docker run -it --rm hello-world

Ensure Python3 is installed and in PATH

python3 -c "print('Confirmed python3 installed.')"

Ensure virtualenv is installed

which virtualenv

Set up virtual environment

virtualenv --python=python3 venv
source venv/bin/activate
pip install -r .circleci/test-requirements.txt

Run system testing

Build the image you want to test

docker build -t airflow ./1.10.5/buster

Run system testing

.circleci/bin/test-airflow airflow

The first time you do the build, and the first time you do the system test it will take longer than subsequent runs. The system testing will install the tested versions of CI tools in /tmp/bin (helm, kubectl, kind). It will leave an airflow cluster running on your kind cluster in 'test-cluster'. When you run it again, it will delete the namespace of your most recent deployment and redeploy into a new namespace. If you make changes in the image, don't forget to re-build the image before testing it.

Use the newly installed tools

export PATH=/tmp/bin:$PATH

Ensure kubectl configured to use kind

kubectl cluster-info --context kind-test-cluster

Look at the pods

kubectl get pods --all-namespaces

Clean up

kind delete cluster --name test-cluster

Scheduled Tasks

The regularly scheduled tasks are:

Edge Builds

• Rebase the astro-main branch of astronomer/airflow onto main and push it to astronomer/airflow:astro-main (this then kicks off a GitHub Actions workflow that builds Airflow and Astronomer Certified Python packages/wheels) and pushes them to our PyPI package repository
• Build nightly Docker images for QA (using those nightly Airflow and Astronomer Certified wheels) and push them to the dev image repository

CircleCI Schedules

The CircleCI documentation on scheduled pipelines is very new and in slight disarray.

See also:

• The announcement for scheduled pipelines
• This relevant (for now) PR, which includes an example of branch filtering
• Documentation on parameter syntax
• Documentation on logic statements

The API for manipulating schedules is documented here, including examples.

Here is an example listing all schedules with HTTPie (and colorizing the response with jq):

$ http https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \
       circle-token:<CIRCLECI_PERSONAL_ACCESS_TOKEN> \
       | jq

To create a new schedule (refer to the HTTPie docs about raw JSON):

$ http --verbose \
       https://circleci.com/api/v2/project/gh/astronomer/ap-airflow/schedule \
       circle-token:<CIRCLECI_PERSONAL_ACCESS_TOKEN> \
       name="every-morning-0200-UTC" \
       description="Every morning at 02:00 UTC" \
       attribution-actor="system" \
       parameters:='{ "branch": "master" }' \
       timetable:='{ "per-hour": 1, "hours-of-day": [2], "days-of-week": ["SUN", "MON", "TUE", "WED", "THU", "FRI", "SAT"]}'

Note that updating and deleting schedules uses a different URL path:

$ http PATCH \
       https://circleci.com/api/v2/schedule/<schedule_uuid> \
       circle-token:<CRCLECI_PERSONAL_ACCESS_TOKEN> \
       name=every-sunday-0100-utc \
       description="Every Sunday at 01:00 UTC"

You can create a CircleCI personal API token in your CircleCI user settings. Do note that a PAT will authenticate as you, and have full, read and write access on CircleCI, so keep your PAT secret and do not publish it anywhere!

License

Apache 2.0 with Commons Clause