search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
774 stars 170 forks source link

Enable Azure task in the remote store manifest example DAG #1333

Closed pankajkoti closed 5 days ago

pankajkoti commented 6 days ago

During the work on PR https://github.com/astronomer/astronomer-cosmos/pull/1297, an issue arose where the Azure remote manifest task began failing after installing providers and packages with constraints. To allow other tests, which were running successfully, to proceed, the task was temporarily disabled. Upon reviewing the GitHub Actions logs from previous successful runs, it was observed that the Azure provider version installed was 10.5.1. However, after the refactoring introduced in the PR, the failing actions showed azure provider==8.4.0 being installed with the constraints file.

To investigate, I tested locally to identify a working version. While the task failed with 8.4.0, it succeeded with 8.5.0. Analyzing the failure logs and reviewing the Azure provider changelog hints that https://github.com/apache/airflow/pull/35820/ is potentially the fix for the failure with our connection setup in the CI that was released in 8.5.0.

Therefore, I propose using azure provider>=8.5.0. However, the Airflow constraints file for version 2.8 specifies azure provider==8.4.0, which conflicts with this requirement. To address this, I am making changes to the pre-install script in our CI to install azure provider>=8.5.0 without relying on the constraints file, citing the reasons above.

closes: #1304

cloudflare-workers-and-pages[bot] commented 6 days ago

Deploying astronomer-cosmos with  Cloudflare Pages  Cloudflare Pages

Latest commit: cc01f68
Status: ✅  Deploy successful!
Preview URL: https://dfec4b03.astronomer-cosmos.pages.dev
Branch Preview URL: https://enable-remote-manifest-store-x3ly.astronomer-cosmos.pages.dev

View logs

codecov[bot] commented 6 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96.03%. Comparing base (573a90a) to head (cc01f68). Report is 2 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1333 +/- ## ======================================= Coverage 96.03% 96.03% ======================================= Files 67 67 Lines 3988 3988 ======================================= Hits 3830 3830 Misses 158 158 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features: