astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
20 stars 10 forks source link

Support templating in the field `notebook_params` of the `DatabricksNotebookOperator` #33

Closed tatiana closed 1 year ago

tatiana commented 1 year ago

Allow the argumentnotebook_params of the DatabricksNotebookOperator to be templated using Jinja. Rendered the attribute before sending it to the Databricks API.

Improve logging and sharing the Databricks API job Xcom across jobs.

Limitation: templating may not work as expected in Airflow 2.3, but it works on Airflow 2.2.5, Airflow 2.4 and Airflow 2.5.

Example of how a Databricks workflow generated using the DAG example_databricks_workflow looks like in their interface (the parameter "ds", passed to the notebook, and "ts", passed to the task group, are resolved to the current datetime timestamp):

Screenshot 2023-04-25 at 06 41 08

Closes: #32

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 74.07% and project coverage change: +0.14 :tada:

Comparison is base (5eefd91) 78.32% compared to head (401ed6a) 78.47%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #33 +/- ## ========================================== + Coverage 78.32% 78.47% +0.14% ========================================== Files 5 5 Lines 443 460 +17 Branches 52 57 +5 ========================================== + Hits 347 361 +14 - Misses 76 78 +2 - Partials 20 21 +1 ``` | [Impacted Files](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [src/astro\_databricks/operators/notebook.py](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL25vdGVib29rLnB5) | `85.34% <41.66%> (-4.28%)` | :arrow_down: | | [src/astro\_databricks/operators/workflow.py](https://codecov.io/gh/astronomer/astro-provider-databricks/pull/33?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL3dvcmtmbG93LnB5) | `86.56% <100.00%> (+3.10%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.