This repo is a quick spike to evaluate use of helm
by the Climate Action Secretariat.
All code is derived from stable charts from the Helm Hub.
Get helm
via your favorite package manager before proceeding.
Try deploying Apache Airflow.
There's no recursive dependency management in helm yet, so deps have to be installed in order:
helm dep up postgresql
helm dep up airflow
helm dep up cas-airflow
helm install --namespace "wksv3k-tools" cas-airflow cas-airflow
helm upgrade --atomic --timeout 600s --namespace "wksv3k-tools" cas-airflow cas-airflow
helm delete cas-airflow
The Kubernetes Executor
runs individual sandboxed pods for each Airflow task & requires some extra setup.
pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh,gcp,kubernetes]==1.10.10
which covers the necessary dependencies.Kubernetes Executor
is sandboxed, it requires the explicit passing of variables to the pod via KUBERNETES_ENVIRONMENT_VARIABLES
. https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg#L1072
config
in values.yaml:AIRFLOW__CORE__REMOTE_LOGGING
: True
AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES__AIRFLOW__CORE__REMOTE_LOGGING
: True
extraEnv
rather than config
.