This PR adds an optional initialization argument to the Ed-Fi Resource DAGs: full_refresh_dom. This adds a new user-defined-macro to the DAG that is referenced in airflow_util.is_full_refresh. When the day of the DAG run matches this variable, the full-refresh parameter will default to True, regardless of DAG-level configs.
PR Merge Priority:
[x] Low
[ ] Medium
[ ] High
Changes to existing files:
edu_edfi_airflow/dags/dag_util/airflow_util.py: Add is_dom() helper to compare the DAG run logical date with the provided dom; overload is_full_refresh() to check for this macro first before returning the param value.
edu_edfi_airflow/dags/edfi_resource_dag.py: Add the full_refresh_dom argument and is_dom_full_refresh UDM to EdFiResourceDAG.
Feature: Day-of-Month Full Refresh
Description & motivation
This PR adds an optional initialization argument to the Ed-Fi Resource DAGs:
full_refresh_dom. This adds a new user-defined-macro to the DAG that is referenced inairflow_util.is_full_refresh. When the day of the DAG run matches this variable, the full-refresh parameter will default to True, regardless of DAG-level configs.PR Merge Priority:
Changes to existing files:
edu_edfi_airflow/dags/dag_util/airflow_util.py: Addis_dom()helper to compare the DAG run logical date with the provideddom; overloadis_full_refresh()to check for this macro first before returning the param value.edu_edfi_airflow/dags/edfi_resource_dag.py: Add thefull_refresh_domargument andis_dom_full_refreshUDM to EdFiResourceDAG.