Open champion1Ping opened 10 months ago
the server is implemented in go language
In general, when the gRPC server get restarted, the client should be mostly intact, the underlying channel could be fall into an unavailable status, and depending on the client configuration, the error could be propagated to the user of the client or get silently retried.
Can you share your client configuration with more details? I'm new to the gRPC-Spring implementation, but this could lead to an area we can improve, with the default error handling & retry logic.
Have the same problem using both server and client implemented in Java. If the server has some temporary problem, when calling from the client I get io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Even if the server recovers from this failure i have to restart the client, otherwise the client gives the same error without calling the server again.
Upon further research it has to do with connection backoff described here https://github.com/grpc/grpc/blob/v1.32.0/doc/connection-backoff.md
Haven't found a way to disable this behaviour
You might want to add a GrpcChannelConfigurer. Although I'm not sure which option you need to change those. But you could ask that upstream or on StackOverflow.
In general, we have 2 layers need to cooperate on temp server failures.
Things missing here is to have a solid RPC retry strategy, thus the underlying connection issue can be hidden from the application layer. Unavailable error is typically retryable, but the application still need to be designed in a way to carefully distinguish idempotent and non-idempotent RPCs to avoid unsafe retries.
status=FAILURE, cause=[io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure