astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
21 stars 11 forks source link

No longer require XCom pickling for 2.2.5 #15

Closed dimberman closed 1 year ago

dimberman commented 1 year ago

In the current implementation, we pass a "DatabricksMetadata" object via Xcom that contains basic metadata like run id and conn id. When supporting earlier versions of Airflow, this would mean that users have to turn on xcom pickling (which would be a security risk). To avoid this risk we simply return a json dictionary that we can then manually deserialize as needed.

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 90.00% and project coverage change: +0.10 :tada:

Comparison is base (c7c1707) 78.31% compared to head (3ac08ba) 78.41%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #15 +/- ## ========================================== + Coverage 78.31% 78.41% +0.10% ========================================== Files 5 5 Lines 415 417 +2 Branches 47 47 ========================================== + Hits 325 327 +2 Misses 73 73 Partials 17 17 ``` | [Impacted Files](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [src/astro\_databricks/plugins/plugin.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3MvcGx1Z2lucy9wbHVnaW4ucHk=) | `68.62% <75.00%> (+0.15%)` | :arrow_up: | | [src/astro\_databricks/operators/notebook.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL25vdGVib29rLnB5) | `92.47% <100.00%> (+0.08%)` | :arrow_up: | | [src/astro\_databricks/operators/workflow.py](https://codecov.io/gh/astronomer/astro-providers-databricks/pull/15?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-c3JjL2FzdHJvX2RhdGFicmlja3Mvb3BlcmF0b3JzL3dvcmtmbG93LnB5) | `83.76% <100.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.