astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
21 stars 11 forks source link

Add support for a matrix of Airflow versions #4

Closed pankajkoti closed 1 year ago

pankajkoti commented 1 year ago

While testing the Databricks workflow and notebook operators across various versions of Airflow, it was observed that quite a few attributes are not available as per their current usage available in the latest Airflow version. To support these limitations with older Airflow versions, the PR adds the following fixes:

  1. DatabricksNotebookOperator: The task_group attribute is not available with the BaseOperator. For that, we fetch the task group from the TaskGroupContext
  2. DatabricksWorkflowTaskGroup: Like in DatabricksNotebookOperator, we need to fetch the task group from the TaskGroupContext and additionally, the dag attribute is not available with the TaskGroup base class which we need to fetch from the DagContext
  3. plugin.py module: The airflow_flask_app utility is unavailable. So, we use the current_app from the flask context to get the Airflow flask instance. Additionally, the task_group is not available in the operator. So, we have added a custom implementation to fetch the task group for the legacy Airflow version with also fixing the way how we construct URLs for the views.
  4. conftest.py: The EmptyOperator used here is unavailable. So we make use of the DummyOperator here
  5. dag.test() is unavailable. So, we introduce local utilities to leverage the test_dag utility for us to run example DAGs without the scheduler

Additionally, in order to support running the example DAGs i.e. running concurrent Databricks jobs with the same notebooks across multiple Airflow and Python versions we have introduced the following changes:

  1. Provider unique job_cluster_key to the Databricks jobs
  2. Add the Airflow (2.2.4, 2.3, 2.4, 2.5) and Python (3.8, 3.9) versions for the matrix in ci.yml

closes: #3

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 45.37% and project coverage change: -11.29 :warning:

Comparison is base (c7f4e2a) 89.60% compared to head (8ffd25a) 78.31%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #4 +/- ## =========================================== - Coverage 89.60% 78.31% -11.29% =========================================== Files 5 5 Lines 327 415 +88 Branches 31 47 +16 =========================================== + Hits 293 325 +32 - Misses 24 73 +49 - Partials 10 17 +7 ``` | [Impacted Files](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/4?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [src/astro\_databricks/operators/workflow.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/4?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL3dvcmtmbG93LnB5) | `83.76% <20.00%> (-6.06%)` | :arrow_down: | | [src/astro\_databricks/operators/notebook.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/4?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL25vdGVib29rLnB5) | `92.39% <25.00%> (-3.12%)` | :arrow_down: | | [src/astro\_databricks/plugins/plugin.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/4?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3MvcGx1Z2lucy9wbHVnaW4ucHk=) | `68.47% <48.93%> (-16.57%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.