Use GracefulStop instead of Stop for closing grpc connections on frontend

[dhiaayachi]

Is your feature request related to a problem? Please describe.

I would like temporal to drain active connections while refusing new connections

Describe the solution you'd like

using grpc GracefulStop with the shutdownDrainDuration as a timeout before focefully closing connections seems like a good option

Describe alternatives you've considered

since we use long poll we probably can't actually block until connections close and need to tell clients to go away after a time. any approach to not accepting new connections, waiting a drain duration and then sending go away before finishing shutdown would work.

Additional context

this could help reduce disruption to workers during upgrade especially for clients that are sensitive to connections just going away.

[dhiaayachi]

Thank you for your feature request! This is an interesting idea that could benefit Temporal users.

While Temporal doesn't currently support graceful shutdown with connection draining, you could potentially implement a custom solution using the existing Temporal API. Here's a possible workaround:

1. Signal Workers: When starting the shutdown process, signal your workers with a specific signal (e.g., "shutdown-pending").
2. Client Logic: Have your clients handle this signal and stop sending new requests to Temporal.
3. Timeout: Introduce a grace period within the workers to allow for existing tasks to complete.
4. Close Connections: After the grace period, initiate a clean shutdown, gracefully closing existing connections.

This approach allows you to manage client behavior during shutdown and minimizes disruption.

Keep in mind this is a custom solution and might require careful implementation to ensure proper handling of in-flight requests and potentially long-running tasks.

If you need further assistance or have additional questions, please don't hesitate to ask. We're here to help!

[dhiaayachi]

Thank you for your feature request!

This is a great idea for reducing disruption during upgrades. Currently, Temporal doesn't have built-in functionality for graceful shutdown with connection draining.

Here are some suggestions for working around this:

• Use a separate load balancer: You can use a load balancer to direct new traffic away from the shutting down Temporal cluster. This way, Temporal can gracefully close existing connections while refusing new ones.
• Implement custom logic: You can implement custom logic in your application to handle the graceful shutdown process. For example, you could introduce a flag to indicate that the service is shutting down and have your clients react accordingly.

We'll keep your feedback in mind for future development!