[source-freshservice] API error 429 when sync requesters stream

Connector Name

Freshservice

Connector Version

v1.3.0

What step the error happened?

During the sync

Relevant information

When we try to synchronize the "requesters" stream, we are receiving error 429 which I believe has to do with some API limit.

The error occurs close to page 130 and by testing with direct calls through the API, we validate that this object has more than 1200 pages.

Relevant log output

2024-02-05 12:41:54 source > Request to https://MYCOMPANY.freshservice.com/api/v2/requesters?page=137 failed with status code 429 and error message None
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module></module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 213, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 120, in run
    yield from map(AirbyteEntrypoint.airbyte_message_to_string, self.read(source_spec, config, config_catalog, state))
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 162, in read
    yield from self.source.read(self.logger, config, catalog, state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 134, in read
    yield from super().read(logger, config, catalog, state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 143, in read
    raise e
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 123, in read
    yield from self._read_stream(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 195, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 255, in _read_full_refresh
    for record_data_or_message in stream_instance.read_full_refresh(configured_stream.cursor_field, logger, self._slice_logger):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 121, in read_full_refresh
    yield from self.read_records(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 104, in read_records
    yield from self.retriever.read_records(stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 304, in read_records
    for stream_data in self._read_pages(self._parse_records, self.state, stream_slice):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 282, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 256, in _fetch_next_page
    return self.requester.send_request(
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 413, in send_request
    return self._validate_response(response)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 518, in _validate_response
    raise ReadException(error_message)
airbyte_cdk.sources.declarative.exceptions.ReadException: Request to https://MYCOMPANY.freshservice.com/api/v2/requesters?page=137 failed with status code 429 and error message None

Contribute

[ ] Yes, I want to contribute

airbytehq / airbyte