Chaining rate controllers

[corentin-regent]

Sometimes, the ability to chain buckets does not suffice, and we would need to chain rate controllers. For example let's say that we work with multiple REST APIs at the same time, each of them using their own rate controller.

We can already chain a global bucket with each one of these buckets, using BucketGroups, for example if we would like globally to not send too many requests at once. However, if we would like to enfore global constraints, such as defining a global concurrency limit, well currently we cannot.

A solution, that has yet to be implemented, would be to share a global rate controller with no bucket but only this concurrency limit, and chain each of the individual rate controllers with it.

[corentin-regent]

It seems to me that implementing such combination of rate controllers in an efficient way is not trivial, and would require deep changes in the way Rate Control works.

I have searched other projects for ideas on how to achieve such feat, however I could not find any that actually did. The closest I could find was the popular Bottleneck library for JavaScript, but the implementation is quite disappointing as it is equivalent in Rate Control to the following:

async with limiter_a.request(...), limiter_g.request(...):
    ...  # perform the API call

Such solution is not acceptable for us, as limiter_a accepts the request before limiter_g even receives it, inducing an unpredictable lag between the requests limiter_a keeps track of and the ones that are eventually sent to the API.

So I guess that something we can do in Rate Control, to be able to enforce both fine-grained and global constraints, is use some kind of trackers in the rate controllers, that may be shared between them similarly to what can be done with buckets.

The rate controllers then could simply increment the statistics of their shared trackers on each request, and it would become the tracker's job to raise errors when limits are reached.

I may reconsider this solution though as it does not seem very beautiful; I would be happy to hear other ideas!