This is now performed with an Airflow PythonOperator which runs the
clone_or_update_pipeline function in utils.py. This will update a local
copy of a pipeline if it already exists, and clone it if not. That function
is intended to be idempotent if the ref is a tag or commit, or if the
remote branch (e.g. origin/master) hasn't changed.
To allow importing from utils.py, make the dags directory into a Python
package by adding an empty __init__.py.
I'm not familiar with how Airflow imports from the Python files in dags,
so I'm not very confident that this will work as-is, but I'm not yet set up
to test Airflow myself.
This is now performed with an Airflow PythonOperator which runs the
clone_or_update_pipelinefunction inutils.py. This will update a local copy of a pipeline if it already exists, and clone it if not. That function is intended to be idempotent if the ref is a tag or commit, or if the remote branch (e.g. origin/master) hasn't changed.To allow importing from
utils.py, make thedagsdirectory into a Python package by adding an empty__init__.py.I'm not familiar with how Airflow imports from the Python files in
dags, so I'm not very confident that this will work as-is, but I'm not yet set up to test Airflow myself.