The purpose of this template is to easily deploy Apache Airflow on Openshift.
The used Airflow container image is built using the official image source codes distributed by Airflow, built with the additional providers packages by Apache Spark, Papermill and Mongo. It should also be possible to build your own image and deploy the template using that image, as long as the image is built using the official sources.
The template can be imported via the Openshift web UI or added via the command line interface (oc)
The required variables which need to be present in order to run the template are:
Rest of the variables are optional and have a value by default which can be changed, if needed.
Some of the useful variables to you as the user could be:
The current template deploys a Jupyter pod for writing the python code for DAGs. The password will be the one you set in JUPYTER_PASSWORD variable. Note When accessing Jupyter, you need to click on Upload to upload an existing python code (.py extension) of the Airflow DAG or you could click on New->Text File and then write the python code in the text file itself, but remember to save it with the .py extension
It can take up to 5 minutes for the DAGs to show up in the web UI, so be patient!
The most reliable way to include your DAG dependencies in the image is to build your own Airflow image using the official sources, and include the dependencies in the building phase of the image. More information about building the image can be found here. After building the image, push it to some image registry, and set the image link to the AIRFLOW_IMAGE variable to deploy the template using that image.
You might also want to consider using PythonVirtualenvOperator, that creates a virtual environment for a task with the required pip packages, and tears it down after task completion. For more information about the PythonVirtualenvOperator, see the official documentation here and here.
There is also an easy and fast way to install any python libraries when deploying the template, but it is error-prone and only recommended for testing. To use the easy method, you can:
Before Deployment: Use the variable PIP_REQUIREMENTS, where you can specify the name of the libraries separated by whitespace, for example pandas scipy==1.5.4
After Deployment: Edit the configmap pip-requirements and add your requirements there, similarly separated by whitespace. NOTE - when using this option, you need to redeploy the scheduler and worker deployments for the changes to take place!
However, this is a fragile and unrecommended way to install the dependencies, and may result in error.
If you want to change the Airflow configuration, the best way to do it is to add new environment variables in the deployment configs of the pods. Be aware that some variables have to be set in the worker pods, while some have to be set in the webserver pod for the effect to take place! For more information about configuring Airflow with environment variables, check the official documentation here. For a list of all available Airflow configurations, see here.
Airflow can be configured to send emails: you can both send custom emails through Airflow as a task, or receive alert emails yourself if one of your DAG runs have failed. For the email system to work the following configuration variables have to be set in the deployment config of worker:
And, if the smtp host requires it:
If you need CSC specific configuration, contact servicedesk@csc.fi.
To use a Google Gmail account as the email host, you first have to create an App Password to your account. To set up an App Password, follow instructions in https://support.google.com/accounts/answer/185833?hl=en.
When you have the password, enter these as environment variables in the worker deployment config:
Create a connection via the Airflow web UI by clicking on Admin->Connections , then fill the following fields:
{"aws_access_key_id":"your-access-key-id", "aws_secret_access_key": "your-secret-access-key", "host": "the-s3-endpoint-url"}