This PR introduces support for DAG Factory YAML configuration files to support referencing environment variables, such as $SOME_PATH. Now DAG Factory resolves this during DAG parsing time.
Motivation
Troubleshooting DAG Factory, for example, was difficult because DAGs had all hard-coded paths based on the Dockerfile.
The following commands failed to run because of the hard-coded paths:
PYTHONPATH=`pwd`:`pwd`/examples AIRFLOW_HOME=`pwd` airflow dags test example_breadfast
PYTHONPATH=`pwd`:`pwd`/examples AIRFLOW_HOME=`pwd` airflow dags test test_expand
Alternatives
Since DAG Factory expected absolute paths, using relative paths was not an option.
What changes
By setting the environment variable CONFIG_ROOT_DIR to reference the root directory where DAG Factory example DAGs configs are, anyone should be able to run DAG Factory locally using the airflow dags test command:
CONFIG_ROOT_DIR=`pwd`"/dags" PYTHONPATH=`pwd`:`pwd`/examples AIRFLOW_HOME=`pwd` airflow dags test test_expand
This PR introduces support for DAG Factory YAML configuration files to support referencing environment variables, such as
$SOME_PATH. Now DAG Factory resolves this during DAG parsing time.Motivation
Troubleshooting DAG Factory, for example, was difficult because DAGs had all hard-coded paths based on the
Dockerfile.The following commands failed to run because of the hard-coded paths:
Alternatives
Since DAG Factory expected absolute paths, using relative paths was not an option.
What changes
By setting the environment variable
CONFIG_ROOT_DIRto reference the root directory where DAG Factory example DAGs configs are, anyone should be able to run DAG Factory locally using theairflow dags testcommand: