[Enhancement]: Add standardized GRPC Healthcheck endpoint

[pregenRobot]

Is there an existing issue for this?

• [X] I have searched the existing issues

What would you like to be added?

I would like to add an endpoint to perform healthcheck using the standardized GRPC Healthcheck endpoint

https://github.com/grpc/grpc/blob/master/doc/health-checking.md

Why is this needed?

Software load balancers often require us to perform healthchecks before requests can be routed to the service in a Kubernetes cluster. It becomes a big hassle if an unstandardized health checking endpoints for a service is provided, instead of the standardized one as we will have to implement a separate logic specifically for Milvus.

Anything else?

In order to provide compatibility and prevent breaking changes, we propose a separate healthcheck endpoint (aside from the existing one) that follows the standardized GRPC health checking protocol.

[congqixia]

I'm a little bit lost about the background of this need. Milvus do has CheckHealth grpc service but not fully compatible with grpc standard one. And team is working on enhancing the meaning of this CheckHealth API. If could be confusing if these both of these two services exist in Milvus proto. I was wondering that if you could elaborate the detail need for this API please? @pregenRobot /cc @jaime0815

[pregenRobot]

@congqixia @jaime0815

Thank you for responding back!

To provide some context behind why we want this feature:

• We run an internal NGINX load balancer to handle traffic coming via the internet and relaying it to our Milvus service running on Kubernetes.
• The load balancer supports HTTP requests and GRPC requests. Regardless of HTTP or GRPC, a health check must be conducted before any traffic gets routed to Milvus.
• In the case of HTTP, an HTTP based health check must be conducted. Milvus provides /healthz at port 9091 so our load balancers can route HTTP requests to Milvus
• But in the case of GRPC, the standard GRPC healthcheck endpoint must be made available. Because Milvus does not provide the standardized healthcheck RPC and instead has a custom one, the loadbalancer's health check fails and thinks that the Milvus service is not available. Therefore, no GRPC traffic gets routed to Milvus.
• Milvus python SDK uses GRPC internally so therefore, we cannot use the Python SDK to interact with Milvus via the internet, accessing it through NGINX.

The solution to add the standardized GRPC Healthcheck endpoint we believe is the best way to resolve this issue. Why?

• Previously Milvus used to support the standardized GRPC Healthcheck (https://github.com/milvus-io/grpc-milvus/blob/master/src/proto/grpc/health/v1/health.proto#L42) so this is just re-introducing an old feature
• We believe we are not the only ones who have a usecase where the standardized grpc health checking endpoint will be necessary due to constraints of NGINX or kubernetes NGINX plugins. Milvus will be a much easier service to run and maintain if it follows more standardized protocols.

I will appreciate your feedback on what you think about this proposal! I understand the team is working on expanding the CheckHealth API and feel it would be really valuable if the extended functionality you are considering takes into account the standard.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.