API Rate Limit Exceeded on ZenDesk connector

[anatolec]

## Environment - **Airbyte version**: 0.35.4-alpha - **OS Version / Instance**: AWS EC2 t2.large - **Deployment**: Docker - **Source Connector and version**: Zendesk 0.2.0 - **Destination Connector and version**: irrelevant - **Severity**: High - **Step where error happened**: Sync job ## Current Behavior The Zendesk connector randomly fails due to an API Rate limite exceeded error ## Expected Behavior Query the zendesk API slowly enough so that it does not happen. Or allow to configure several API tokens to balance the quota consumption (as it is possible in the GitHub connector) ## Logs

LOG

``` 2022-03-10 03:30:58 source > [Errno Expecting value] Number of allowed API requests per minute exceeded: 0 Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in launch(source, sys.argv[1:]) File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 127, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 118, in run for message in generator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 123, in read raise e File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 119, in read internal_config=internal_config, File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 159, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py", line 215, in _read_incremental for record_counter, record_data in enumerate(records, start=1): File "/airbyte/integration_code/source_zendesk_support/streams.py", line 246, in read_records self.generate_future_requests(sync_mode=sync_mode, cursor_field=cursor_field, stream_slice=stream_slice, stream_state=stream_state) File "/airbyte/integration_code/source_zendesk_support/streams.py", line 185, in generate_future_requests records_count = self.get_api_records_count(stream_slice=stream_slice, stream_state=stream_state) File "/airbyte/integration_code/source_zendesk_support/streams.py", line 174, in get_api_records_count records_count = response.json().get("count", {}).get("value", 0) File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 917, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: [Errno Expecting value] Number of allowed API requests per minute exceeded: 0 ```

## Steps to Reproduce 1. Set up a Zendesk connection 2. Launch syncs until it fails (it's very random...) ## Are you willing to submit a PR? Not at the moment

[anduresu]

Update: I have the same problem, I found out that this problem happens with the Zendesk Support Connector on Version 0.2.0 onwards and 0.1.12 works correctly. The following log shows the same problem OP has, I used S3 as destination.

• Version 0.2.1 = logs-3.txt.
• Version 0.2.0 = logs-14.txt
• Version 0.1.12 = logs-12.txt

Additionally, Version 0.1.12 of Zendesk Support Connector finds around 9k records for 1 day and for the same amount of time, Version > 0.2.0 finds around 900k records, which seems wrong for the usage amount.

[lgomezm]

I'm also seeing that rate-limit issue. I have the following setup:

• One single Zendesk Support source.
• One connection from that source to Google Pub/Sub, runs every 15 minutes.
• Another connection from that source to Google Pus/Sub, runs every 2 hours.

At some point, the rate-limit issue occurs in either of the jobs of those connections, as it can be seen in these logs: logs-364157.txt logs-365810.txt

[bazarnov]

@AnatolePledg, @anduresu, @Igomezm Hello, general recommendation to isolate the issue is to try to update to 0.2.2 version, make the full-refresh to some test place, like Local CSV or Local Json and share the logs here, please. We've slightly changed the way some streams work, such as: TicketComments. The change decreases the amount of API calls.

[bazarnov]

@AnatolePledg @anduresu @lgomezm Please reply to previous message once you test the 0.2.2 version.

[anatolec]

Hi @bazarnov , I haven't faced this issue for many days now, at least since my upgrade to 0.2.2 a week ago. I will close this issue in a week if the problem hasn't reappeared

[bazarnov]

@AnatolePledg Got it. Sounds great, moving this issue to On Hold for now.

[anduresu]

Hi @bazarnov , I updated the connector and works correctly. Thanks

[bazarnov]

Closing this issue, due to no actions required. Otherwise, please reopen it again.

[danieldiamond]

@bazarnov I am experiencing this issue. I am unable to sync all tickets and am hitting this limit every time. Unable to even pull a few days worth