search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.76k stars 3.79k forks source link

[source-posthog] increase exponential backoff #37334

Open meyerovb opened 2 months ago

meyerovb commented 2 months ago

Connector Name

source-posthog

Connector Version

1.0.0

What step the error happened?

During the sync

Relevant information

Default backoff configuration insufficient

Relevant log output

2024-04-16 02:46:51 platform > Records read: 120000 (21 MB)
2024-04-16 02:46:52 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 2202 seconds.","attr":null})
2024-04-16 02:46:52 source > Retrying. Sleeping for 10 seconds
2024-04-16 02:47:03 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 2191 seconds.","attr":null})
2024-04-16 02:47:03 source > Retrying. Sleeping for 20 seconds
2024-04-16 02:47:24 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 2169 seconds.","attr":null})
2024-04-16 02:47:24 source > Retrying. Sleeping for 40 seconds
2024-04-16 02:48:05 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 2128 seconds.","attr":null})
2024-04-16 02:48:05 source > Retrying. Sleeping for 80 seconds
2024-04-16 02:49:26 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 2047 seconds.","attr":null})
2024-04-16 02:49:26 source > Retrying. Sleeping for 160 seconds
2024-04-16 02:52:07 source > Giving up _send(...) after 6 tries (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://app.posthog.com/api/projects/34289/persons?offset=120000&limit=100, Response Code: 429, Response Text: {"type":"throttled_error","code":"throttled","detail":"Request was throttled. Expected available in 1886 seconds.","attr":null})
2024-04-16 02:52:07 source > Max retry limit reached. Request: <preparedrequest>, Response: <response>
2024-04-16 02:52:07 source > Encountered an exception while reading stream persons

Contribute

marcosmarxm commented 2 months ago

Thanks for reporting the issue @meyerovb Posthog is a community connector and it isn’t in the current roadmap for improvements. If you want to contribute fixing the issue please reach me out in Slack so I can provide you instructions to make the contribution