nataliekwong
commented
1 year ago @sherifnada Would this fit in scope for the API connectors team or Connector Ops? This would primarily be relevant for historical syncs of large sources. In this case, the source is an external-facing application so the risk of breaching the limit is application downtime.
This feature as proposed creates a limit or allocation on the user level, which Salesforce doesn't support. In this specific situation, a less explicit way to approach this could be to always stop syncing if the instance is above a percentage threshold of the limit rather than wait for the limit to be reached, which (for Salesforce) is returned in the response header (REST) or body (SOAP).
What area the feature impact?
Connectors
Revelant Information
For all sorts of reasons, a user may want to arbitrarily limit the requests that Airbyte is making to the source connector. I suggest that an ability be provided for the user to limit requests to X number every Y period, where period could be minute or hours.
The use case for me was in extracting data from Salesforce. When Salesforce API thresholds are breached, they cut off API access completely and this impacts our Production application. However, I can see this being useful in other cases where throttling Airbyte requests can maintain Production performance in the source.