grpc-ecosystem / grpc-spring

Spring Boot starter module for gRPC framework.
https://grpc-ecosystem.github.io/grpc-spring/
Apache License 2.0
3.43k stars 809 forks source link

When grpc client is deployed in Kubernetes, what is the usecase of HealthIndicator (actuator/health) that reports the service as OUT_OF_SERVICE if any client has the TRANSIENT_FAILURE status? #1022

Open matevskial opened 6 months ago

matevskial commented 6 months ago

The context

I have deployed microservice on Kubernetes that has configured liveness probe that uses the spring actuator's endpoint to check the health of microservice. If the livenesss probe returns status code greater than 400, then kubernetes will trigger a pod restart.

The question

What is the usecase of HealthIndicator (actuator/health) that reports the service as OUT_OF_SERVICE if any client has the TRANSIENT_FAILURE status in this context?

Let's say we want to execute a gRPC call with our client to some gRPC server and lets say the gRPC server is temporarily unavailable. In the current situation, the client won't be able to reach the server, and grpc client's internal Channel will report a TRANSIENT_FAILURE status to the used GrpcChannelFactory and thus make the client microservice restart when Kubernetes tests the liveness probe.

The solution of course is to disable health integration altogether either by:

Then the question remains, what is the point of this? I just lost my whole day investigating why the microservice was constantly restarting when the microservice tries to call temporarily unavailable grpc server??? Stacktraces and logs

No stactraces and logs are needed.

The application's environment

Which versions do you use?

Additional information

No additional information needed.