search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.15k stars 4.13k forks source link

Source Adwords: Better handling of rate limit exceptions #3517

Closed sherifnada closed 3 years ago

sherifnada commented 3 years ago

Expected Behavior

I expect the Google Adwords source to gracefully retry rate limiting errors.

Current Behavior

A user leveraging a basic access token is reporting that basically all his syncs are failing with a RATE_LIMIT_EXCEEDED error even though this is happening on the first sync he's running. Therefore it seems unlikely that he hit the 10k request/day limit.

Acceptance criteria:

  1. fork the latest version of the Google Adwords tap (done here)
  2. change the tap so that it logs the rateName, rateScope, and retryAfterSeconds fields described here when a rate limit error is reached
  3. change the tap so that it waits for retryAfterSecond * 2 seconds instead of the default 60 second amount. This behavior is recommended by the API docs here. under the retryAfterSeconds header.
  4. Update the Google Adwords Airbyte connector to depend on the forked connector
  5. bump version/update changelog/publish connector

Logs

https://airbytehq.slack.com/files/U01N7JCHS8M/F022H9JJWVB/logs-285-0.txt?origin_team=T01AB4DDR2N&origin_channel=C01ABDJV89H

Steps to Reproduce

This isn't happening in our own CI, but is reported by zestyping . Logs attached above. Slack thread: https://airbytehq.slack.com/archives/C01MFR03D5W/p1621534812097200

Severity of the bug for you

High - user is unable to use the connector

Airbyte Version

latest

Connector Version (if applicable)

0.2.2

┆Issue is synchronized with this Asana task by Unito

zestyping commented 3 years ago

Commenting to follow this issue.

sherifnada commented 3 years ago

@zestyping based on our investigation last week, I was under the impression the rate limit failure here was actually expected since you had exhausted the quota the previous day? Are you seeing failures happening when the rate limit has not been exhausted?

zestyping commented 3 years ago

@sherifnada I believe you're right. Ever since Google upgraded our token to Standard Access we haven't seen rate limit errors (though we do have jobs that run for a very long time — one took 5 hours and the current one is at 15 hours and still running).