Concurrent CDK: Rate limit

[girarda]

What

It should be possible for a source connector to define a credit-based rate limit to avoid busting the API's limit

A simple implementation is to define a limit of N requests for a given time window, and keep a count of the number of request sent during the current window.

• When a worker wants to submit a request:
• If the count is already >= limit: worker cannot submit a request and the worker has to wait
• If the current time is later than the end time of the window, start a new window and reset the count
• Increment the count and allow the worker to submit a request

Note:

• The can_submit_request method needs to be thread-safe because it will be called from multiple workers. It could be protected using a mutex
• This should be implemented in the concurrent CDK
• There are existing python libraries implementing rate limits (eg https://pyratelimiter.readthedocs.io/en/latest/)

Testing

• Write a unit test confirming we're only calling a function X times per second

Acceptance criteria

• [ ] HTTP source connector can define a rate limit in terms of N requests per [time period]
  • Should be configurable for HttpStream
• [ ] The rate can be a fixed constant, supporting dymamic limits can be done in the future

[girarda]

Noting that this feature can be useful in other contexts (eg for vector db destination). If possible, we should implement the rate limiter such that it can be reused outside of the a "concurrent CDK" source

[girarda]

Grooming notes:

• It would be better not to increase the count if the request is already cached

[lucasfcnunes]

I can say that https://docs.airbyte.com/integrations/sources/clickup-api/ could use this and benefit people with the following plans https://clickup.com/api/developer-portal/rate-limits/#:~:text=The%20rate%20limit%20varies%20based,requests%20per%20minute%20per%20token.

Free Forever, Unlimited, and Business Plan: 100 requests per minute per token.

Surely, other APIs have those "plan limits" and I do think it's good practice to limit requests to 3rd party APIs to prevent being confused with a bad bot/IP blocking.