grpc / grpc-dotnet

gRPC for .NET
Apache License 2.0
4.22k stars 777 forks source link

Running server-streaming calls on HealthCheck API delay server shutdown #2572

Open freibenjamin opened 2 weeks ago

freibenjamin commented 2 weeks ago

What version of gRPC and what language are you using?

Grpc 2.46.6
Grpc.AspNetCore.HealthChecks 2.66.0

What operating system (Linux, Windows,...) and version?

Windows 11

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET SDK 8.0.404

What did you do?

We integrated the Grpc.AspNetCore.HealthChecks NuGet package and tested its gRPC API with clients using the Watch server-streaming RPC. To monitor the server's status, we implemented a simple health check class that verifies the server's running state. However, during server shutdown, we noticed a delay if clients were connected to the Watch RPC at the time.

What did you expect to see?

A graceful shutdown that completes within a few seconds at most.

What did you see instead?

When the server stops, the shutdown is significantly delayed if clients are connected to the Watch RPC at the time of shutdown. The still opened calls cause the server to reach its shutdown timeout. We retained the host shutdown timeout at 30 seconds. The screenshot below shows a client connected to the Watch RPC. The shutdown was initiated at 12:07:26 (via the terminal) and completed at 12:07:56. Screenshot 2024-11-15 120818

Anything else we should know?

This issue is related to: https://github.com/grpc/grpc-dotnet/issues/2357

The problem is that running server-streaming calls are not informed about the shutdown. You will find my proposed solution later in the linked PR.