Share experience with the limiter

[elandau]

We'd love to hear about how people are using the limiter. We'd especially like to hear about where it works and under which scenarios it does not perform as expected. Feel free to comment on this issue with your observations.

[cskinfill]

no experience yet ... but i'm wondering how this can or should be used with rxjava2? If we create a scheduler based on this executor, would that work? provide additional back-pressure?

[iqiyiadsrv]

I'am trying to a c++ implementation, I also wonder how many people practiced the limiter, has this limiter practiced in Netflix or some other companies you know? @elandau

[platinummonkey]

A Golang implementation: https://github.com/platinummonkey/go-concurrency-limits

[m-hogue]

@elandau : concurrency-limits works well for heterogeneous workloads for simple requests, but we've run into a situation which may be atypical but isn't well suited for the current implementation.

We have a servlet filter limiter partitioned by REST endpoint(s). Most of our endpoints are simple REST requests, but there are some endpoints that support batching/paging results. As in one logical question spans multiple REST requests in the following fashion:

POST /rest/create
GET /rest/batch
...
GET /rest/batch
POST /rest/close

When concurrency limits are exceeded and requests begin to be rejected across partitions, we could terminate one of these batch-style "requests" at any point in the batching lifecycle. From a client experience perspective, this is pretty bad since arbitrary batch requests could be rejected. It makes sense to reject a single request if the system is overloaded, but this approach complicates the concurrency limits solution.

I could see a couple of approaches to address this:

1. If we had hooks into when a request is going to be failed with a 429 and a handle on the HTTP request itself, then we can clean up any resources stood up by the original /create request and prevent subsequent /batch calls.

2. This collection of requests is one logical question. It'd be nice to prevent new /create requests from coming in, but allow the /batch for already running queries to proceed. I could see this being challenging since the whole point of concurrency limits is to prevent getting overloaded, but avoids the weird behavior where some /batch calls succeed and others could fail.

3. Perhaps a partition priority would be nice, allowing you to say "reject requests in this partition before these other partitions." Again, though, this complicates the rejection strategy and may not actually solve the "stop the bleeding before it gets bad" solution this library provides.

Any thoughts?

[martinlocurcio]

Hi! I've a use case in which the limiter is not working well due to GC Pauses.

https://stackoverflow.com/questions/59311752/how-to-limit-concurrency-in-a-webapp-when-gc-pauses-last-more-than-the-average-r#

All the details are in that question