search

vmagamedov / grpclib

Pure-Python gRPC implementation for asyncio
http://grpclib.readthedocs.io
BSD 3-Clause "New" or "Revised" License
936 stars 92 forks source link

Support for `grpc.max_connection_age_ms` and `grpc.max_connection_age_grace_ms`? #152

Open jalaziz opened 2 years ago

jalaziz commented 2 years ago

We've recently come across the issue where gRPC clients are not correctly load balancing across all available gRPC servers. We are setting the round_robin load balancing policy on the clients, but we have a dynamic environment where servers scale in and out.

While researching this issue, it seems that we must set max_connection_age_ms and max_connection_age_grace_ms on the server to force the clients into re-resolving the DNS hostname and discover new servers when scaling out.

While we could also solve this problem using proxies, it seems like these are settings that grpclib should support to allow for robust client side load balancing.

I realize that this may be blocked on h2 properly handing GOAWAY, but wanted to open this issue in case it's something that could be implemented without GOAWAY support.

vmagamedov commented 2 years ago

For sure client-side load balancing is an important feature.

In order to quickly discover new instances and drain old ones it is better to use special protocol for look-aside load balancing. Do you considered using xDS API instead of relying on DNS and tuning connection age?

jalaziz commented 2 years ago

For sure client-side load balancing is an important feature.

In order to quickly discover new instances and drain old ones it is better to use special protocol for look-aside load balancing. Do you considered using xDS API instead of relying on DNS and tuning connection age?

Indeed we are considering using xDS and/or a service mesh, but it seems like the "fast" path is round_robin load balancing + server connection timeouts.

There are other reasons why these settings can be helpful besides load balancing which is why it might still be worth trying to fix this at some point (e.g. periodic recycling of connections, servers, etc).

ToucanBran commented 2 years ago

@jalaziz how did you set round_robin on the channel object for this library?

ToucanBran commented 2 years ago

@vmagamedov ^^

jalaziz commented 2 years ago

@jalaziz how did you set round_robin on the channel object for this library?

@ToucanBran Currently our clients are Golang clients, but the server is in Python using grpclib. grpc.max_connection_age_ms and grpc.max_connection_age_grace_ms are server settings.

That being said, we are introducing Python clients soon and not having the ability to set the LB policy will end up being an issue.