Use some kind of algorithm (Token Bucket e.g.) to keep track of how many requests per second are happening against certain endpoints. You can use the Databricks API rate limits as an example.
If we are approaching the rate limit, add a delay to the task until more tokens becomes available (if using Token Bucket Algorithm)
Ideally this would be a decorator @rateLimit(key='/'jobs/runs/get'). The key is used as a lookup in some global hash table to keep track of how many requests per second are being issues against the endpoint.
YaroBear
commented
3 months ago
@rateLimit(key='/'jobs/runs/get'). The key is used as a lookup in some global hash table to keep track of how many requests per second are being issues against the endpoint.