astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
329 stars 40 forks source link

Set load_secret True in in Databricks S3 load file DAG #2071

Closed pankajastro closed 6 months ago

pankajastro commented 6 months ago

closes: https://github.com/astronomer/astro-sdk/issues/2070 closes: https://github.com/astronomer/astro-sdk/issues/2068

The default value for load_secret is false and because of that Databricks cluster was not able to access the AWS S3 file. I have changed that to True in this PR and the Airflow task is green now. but not sure if this is because of some recent changes in Databricks.

codecov[bot] commented 6 months ago

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (d4fe295) 89.54% compared to head (f4a3e26) 90.12%. Report is 4 commits behind head on main.

Files Patch % Lines
...on-sdk/src/astro/databases/databricks/api_utils.py 0.00% 1 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2071 +/- ## ========================================== + Coverage 89.54% 90.12% +0.57% ========================================== Files 75 75 Lines 4323 4323 Branches 537 538 +1 ========================================== + Hits 3871 3896 +25 + Misses 355 327 -28 - Partials 97 100 +3 ``` | [Flag](https://app.codecov.io/gh/astronomer/astro-sdk/pull/2071/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [PythonSDK](https://app.codecov.io/gh/astronomer/astro-sdk/pull/2071/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | `90.12% <50.00%> (+0.57%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.