gRPC- NoRouteToHostException

[banandh]

Hi - This is issue is occuring even when there are no deployments on the upstream servers.

My gRPC service is a unary service , Batch application calls my gRPC service using blocking stub.

My gRPC service in turn calls two different gRPC services to retrieve the data (upstream gRPC service 1) and do some calculation using the data retrieved((upstream gRPC service 2)

Since it runs in batch mode, earlier My gRPC service was not able to limit the request that it gets, and it went to unhealthy status so as per the suggestion https://github.com/grpc/grpc-java/issues/10164 from this ticket, Have implemented netflix concurrency limiter at my gRPC service so it allows the request based on the system capacity and for the rest of the other request it throws Server limit reached error.

Upstream services that my GRPC calls has also has limitation so it can handle a subset of request what i get mygRPC service can handle. To apply backoff on the client calls from my application have used same netflix client concurrency limiter. Even after having proper concurrency limiting , I'm getting the NoRouteToHostException. I understand this is something due to overloading the service , is that understanding right or what else are the possibility of getting into this exception?

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: applicationALB/...:443
Caused by: java.net.NoRouteToHostException: No route to host
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

Before I get this exception almost 4Million request processed without any issues, and it's suddenly throwing error. After sometime it works fine. Its not the DNS or connectivity Problem something else is causing this error.

[sanjaypujare]

This was also reported in https://groups.google.com/g/grpc-io/c/umB8apUGJ3I and @ejona86 replied.

[sanjaypujare]

@banandh Pls let us know if we can close this issue.

[banandh]

Yes sure

On Thu, 31 Aug, 2023, 11:34 am sanjaypujare, @.***> wrote:

@banandh https://github.com/banandh Pls let us know if we can close this issue.

— Reply to this email directly, view it on GitHub https://github.com/grpc/grpc-java/issues/10522#issuecomment-1700417072, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF72225DOXZLO4YI2WUGGSTXYASQHANCNFSM6AAAAAA4AHCI44 . You are receiving this because you were mentioned.Message ID: @.***>