Closed pankajkoti closed 1 year ago
Patch coverage: 45.37
% and project coverage change: -11.29
:warning:
Comparison is base (
c7f4e2a
) 89.60% compared to head (8ffd25a
) 78.31%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
While testing the Databricks workflow and notebook operators across various versions of Airflow, it was observed that quite a few attributes are not available as per their current usage available in the latest Airflow version. To support these limitations with older Airflow versions, the PR adds the following fixes:
DatabricksNotebookOperator
: Thetask_group
attribute is not available with theBaseOperator
. For that, we fetch the task group from theTaskGroupContext
DatabricksWorkflowTaskGroup
: Like inDatabricksNotebookOperator
, we need to fetch the task group from theTaskGroupContext
and additionally, thedag
attribute is not available with theTaskGroup
base class which we need to fetch from theDagContext
plugin.py
module: Theairflow_flask_app
utility is unavailable. So, we use thecurrent_app
from the flask context to get the Airflow flask instance. Additionally, the task_group is not available in the operator. So, we have added a custom implementation to fetch the task group for the legacy Airflow version with also fixing the way how we construct URLs for the views.conftest.py
: TheEmptyOperator
used here is unavailable. So we make use of theDummyOperator
heredag.test()
is unavailable. So, we introduce local utilities to leverage thetest_dag
utility for us to run example DAGs without the schedulerAdditionally, in order to support running the example DAGs i.e. running concurrent Databricks jobs with the same notebooks across multiple Airflow and Python versions we have introduced the following changes:
job_cluster_key
to the Databricks jobs2.2.4
,2.3
,2.4
,2.5
) and Python (3.8
,3.9
) versions for the matrix inci.yml
closes: #3