search

astronomer / astro-sdk

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
https://astro-sdk-python.rtfd.io/
Apache License 2.0
329 stars 40 forks source link

Introduce sleep in tests to avoid bigquery rate limits #2110

Closed pankajkoti closed 5 months ago

pankajkoti commented 5 months ago

We're encountering difficulties achieving success with the integration test test_aql_replace_existing_table in our CI environment. The issue stems from Google BigQuery imposing rate limits on table operations—specifically, restricting them to 5 per 10 seconds per table. As a consequence, our CI pipeline fails, thereby hindering the release of version 1.18.0. The rate limits are documented here: https://cloud.google.com/bigquery/quotas#standard_tables

Initially, I contemplated skipping this test in CI. However, implementing a 10s sleep for the problematic BigQuery test appears to circumvent the rate limit error, resulting in a fully successful CI build.

I will cherrypick this PR in the release branch.

codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (aa0e57b) 90.23% compared to head (f71337d) 89.39%. Report is 7 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2110 +/- ## ========================================== - Coverage 90.23% 89.39% -0.85% ========================================== Files 75 75 Lines 4323 4317 -6 Branches 538 538 ========================================== - Hits 3901 3859 -42 - Misses 324 356 +32 - Partials 98 102 +4 ``` | [Flag](https://app.codecov.io/gh/astronomer/astro-sdk/pull/2110/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | Coverage Δ | | |---|---|---| | [PythonSDK](https://app.codecov.io/gh/astronomer/astro-sdk/pull/2110/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer) | `89.39% <100.00%> (-0.85%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.