astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
21 stars 11 forks source link

Add TaskGroup support #29

Closed jlaneve closed 1 year ago

jlaneve commented 1 year ago

closes #27

This PR makes a few small changes to support for Airflow TaskGroups in a Databricks workflow:

I've run the example DAG in Astronomer's Databricks account, and it worked like a charm! Worth noting that Databricks themselves don't have support for TaskGroups/dependency groups, so this was pretty cool to put together.

Also worth noting that in the process of putting this together, I uncovered a bug I filed to be addressed separately.

Screen Shot 2023-04-11 at 12 23 38 AM
codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 76.66% and project coverage change: -0.55 :warning:

Comparison is base (0538602) 78.82% compared to head (1ddcdc5) 78.28%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #29 +/- ## ========================================== - Coverage 78.82% 78.28% -0.55% ========================================== Files 5 5 Lines 425 442 +17 Branches 47 52 +5 ========================================== + Hits 335 346 +11 - Misses 73 76 +3 - Partials 17 20 +3 ``` | [Impacted Files](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/29?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [src/astro\_databricks/operators/notebook.py](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/29?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL25vdGVib29rLnB5) | `89.52% <63.63%> (-3.19%)` | :arrow_down: | | [src/astro\_databricks/operators/workflow.py](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/29?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL3dvcmtmbG93LnB5) | `83.46% <84.21%> (-0.57%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.