search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.06k stars 439 forks source link

proxy: dynamic rate limiting #5707

Closed conradludgate closed 12 months ago

conradludgate commented 1 year ago

Motivation

We can be smart and implement some dynamic algorithms to perform load shedding should it be necessary. Given our reliance on control-plane, if control-plane goes down, so do we. We need to make sure that control-plane doesn't get overloaded.

Implementation ideas

Implement an algorithm like AIMD (Additive increase; multiplicative decrease) in order to have a dynamic load quantifier for control-plane. Increase the load factor on success, decrease the load factor on 5XX error.

Make sure that proxy respects this limiter when making calls to control-plane.

should this limit concurrent requests or requests per second? I think concurrent requests as these calls are synchronous so they should be equivalent. Limiting concurrent requests seems easier to me with a kind of semaphore.

If we use a semaphore approach, we need a semaphore that can remove tokens. I've implemented such a semaphore before: https://github.com/conradludgate/squeeze/blob/main/src/limiter.rs - this repo also has some more ideas for dynamic congestion algorithms

vadim2404 commented 1 year ago

@khanova to finish it this week