Allow specifying maximum number of connections on Server

[ejona86]

This will probably be implemented as part of a larger piece of work to control resources on server-side.

http://stackoverflow.com/questions/37338038/how-to-configure-maximum-number-of-simultaneous-connections-in-grpc

Note that slightly different metrics (like maximum number of streams across the server) may be better; I'm not trying to dictate the exact metric, just the need for this sort of control.

[asif31iqbal]

Thanks for creating this issue.

[rmichela]

What's the status of gRPC resource constraints?

[abaldeva]

+1 for this (C++).

[hehe717]

any updates for this issues?

[ejona86]

No updates. Such a feature should be cross-language, and I confirmed that with the other languages today. I guess it would be helpful to know the more precise use cases (I know use cases for this, but I want to know your reason for asking) and expected behavior when the limit is exceeded. We would also need to decide how "big" of a feature this becomes, since I do think there are various levels of completeness and their associated complexity that the solutions could have.

[Daniel-B-Smith]

I can comment about why we would like it at Datadog for Golang. We're actively working on throttlers to prevent OOMs when a server becomes overloaded. We can go a long way via throttling inside our application, but there is still the risk that memory usage at the OS/connection layer will push us over the limit. Ideally in my opinion, when the server hits too many connections, failed connection attempts will return RESOURCE_EXHAUSTED to the client.

[ejona86]

Ideally in my opinion, when the server hits too many connections, failed connection attempts will return RESOURCE_EXHAUSTED to the client.

This is not feasible. Failed connections cause UNAVAILABLE on the client. To cause RESOURCE_EXHAUSTED we'd have to accept the connection and respond with RESOURCE_EXHAUSTED for each inbound RPC.

[Daniel-B-Smith]

Yeah, that makes sense. Are there separate timeouts for connection establishment and the full request? I would rather establish the connection and fail as early as possible than sit around and timeout.

[ejona86]

@Daniel-B-Smith wrote:

Are there separate timeouts for connection establishment and the full request?

Yes. Connection timeouts and RPC timeouts are separate. An RPC will wait if any connection attempts are in progress, but if the connections are in a known-bad state then the RPC can fail immediately. You can get a rough view with https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md . Roughly speaking (load balancing makes things complicated), RPCs will fail if the Channel is in TRANSIENT_FAILURE.

https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md is for connection establishment. It is supposed to start with a 20 second timeout, and in extreme cases increase as the backoff increases over 20 seconds. But I know Java isn't implementing that; it's more of hard-coded timeouts.

[joseret]

Any news on this? or workaround?

[khsahaji]

Facing the same issue with GoLang. Context is getting cancelled due to max number of parallel connections being hit at gRpc stream server. I need to get that error in client side and handle it but the client right now uses "SendMsg" function and finishes sending message before server is hitting the snag. All this leads to lost messages. How to handle this now ? should the client wait for the number of active connection to go down ? How does it get the number of active parallel connection to gRpc stream server ?

[ejona86]

@khsahaji, open an issue on the grpc-go repo. If you think it helps, you can reference this issue from the issue you create.

[SaxyPandaBear]

Is there any traction on making this feature a reality? I think my current options are:

1. Just allow an unbounded number of connections until the gRPC server under the hood can't serve any more concurrent channels (apologies if I flub any terminology, still an infant with gRPC)
2. Create a server interceptor that uses a global semaphore to track the number of requests and start rejecting things when the semaphore is completely used up

[angoenka]

+1 At Beam we have a use case where we use grpc with python as client and java as server in stream - stream fashion. Each stream requires a reservation of 64MB and release the resources once the scream is closed. This is causing OOM as a single client can create multiple connections in a burt. We want to limit the concurrent streams to limit the resource. We have a PR out which delays the allocation of resources using semaphore here. https://github.com/apache/beam/pull/9647 Can we use a build in way in GRPC to do this?

[ejona86]

@angoenka, for a single client bursting, you can set maxConcurrentCallsPerConnection. Granted, that isn't dynamic so you can't allow a single client to burst higher if other clients aren't using resources.

Your specific case may also not work as well for this issue by itself, since you probably want to limit that one method differently from other methods (not limit for the entire server).

[rmichela]

Netflix’s concurrency limits interceptor is an interesting solution to managing server concurrency. https://github.com/Netflix/concurrency-limits

[bruno-kakele]

I am also interested in this. For instance I have a server that accepts streaming rpcs, but, a single user could spawn multiple connections and kill one of my tasks (due to excessive resource usage and also too many active connections, even ones that don't have an active stream). I am interested in how to guard a rpc server against client abuse, either a malicious client or just because there is a bug. Are there any best practices/recommendations/docs about this? Thanks in advance

[macmv]

Also interested in this. I have a server getting a stream of data, but that would be completely broken if two clients would connect. I don't have any way to limit the number of connections, which makes this difficult.

[ejona86]

@macmv, for application-level limits like that, I think you should have an interceptor that limits the number of concurrent RPCs/clients to one particular service or method. Port-level connection limits are too heavy-handed for your use-case and would not work if you have multiple services, which includes things like health checking, stats retrieval, and debugging services.

Also, in your case of "only one", it may be fair to just implement that in your service directly, since it is quite a specialized restriction.

[mxh1024]

I want to know if this feature is planned or resolved, and what is the reason why it is not currently supported

[jgheewala]

@ejona86 any update on this?