Allow the embedded web server to be shut down gracefully

[LutzStrobel]

We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load. We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.

If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.

[dheerajjoshim]

Hi, I have a spring-boot app with HTTP endpoints and Kafka consumers. I have tested this graceful shutdown implementation and it looks like the whole shutdown is delayed waiting for HTTP with Thread.sleep(), so no other components are notified. This means I need 2*n graceful period if I wanted to give HTTP and Kafka n seconds to finish their thing.

@wilkinsona I saw your comment about not needing SmartLifecycle just for tomcat, but have you considered other components needing graceful shutdown as well? In my case, I would like to leave some time for both @controllers and Kafka Listeners to be able to finish already started processing -- which in some cases can make external calls with exponential backoff, so I would also like to be able to check in retry loops if shutdown was initiated.

I'm thinking about such logic:

• shutdown initiated

• components notified about shutdown, should not block, clock starts

  • HTTP connections no longer accepted
  • kafka consumers stop polling new messages
  • retry loops can early exit, throwing "retry-able" exception
• when time runs out, interrupt processing threads
• finish shutdown, cleanup

Would you consider changing the above logic, or recommend a custom implementation instead?

@gnagy Even in our deployment, we have HTTP endpoints and Kafka consumers. So we want to stop Kafka consumers from consuming the messages and processes already consumed messaged within the grace period. Did you end up implementing a custom method? Or overriding GracefulShutdown implementation?

[gnagy]

@gnagy Even in our deployment, we have HTTP endpoints and Kafka consumers. So we want to stop Kafka consumers from consuming the messages and processes already consumed messaged within the grace period. Did you end up implementing a custom method? Or overriding GracefulShutdown implementation?

I no longer have access to that project but if I remember correctly Spring Kafka integration already supported stop()-ing the listener container when the application context is shutting down, we just had to make sure our custom RetryTemplates in the consumers got notified as well.

[dheerajjoshim]

@gnagy Thanks. I saw spring-Kafka has stop() method on application context shutdown. I think it would be nice to test rolling upgrade under load to see if any messages are lost during shutdown

[zdaccount]

Now there is the graceful shutdown feature (https://docs.spring.io/spring-boot/docs/current/reference/html/web.html#web.graceful-shutdown), but this feature uses a timeout, and only waits within the timeout. But sometimes I want to wait till all the requests are processed, without a timeout. Could you please support this?

[bclozel]

@zdaccount what would happen if one of those outstanding requests is blocked and never finishes? The application instance would then stay up for ever?

[zdaccount]

Yes, sometimes I want that. If the application instance hangs, then I may force kill it, but sometimes I don't want the instance itself to do this.

I think that this is like how the web server processes requests. When the web server process a request, there is no timeout on the process logic (I assume this, but I may be wrong, since I am new to Spring Boot). If this is the case, then it is also reasonable to have no timeout for the last requests.

[wilkinsona]

I don’t think it’s reasonable to have no timeout at all. For one, it is highly unlikely that any network connection will survive indefinitely so the server will be unable to send a response.

Instead, I would recommend configuring a timeout that’s slightly greater than the maximum time that you would wait before manually killing the application instance.

[zdaccount]

I don’t think it’s reasonable to have no timeout at all. For one, it is highly unlikely that any network connection will survive indefinitely so the server will be unable to send a response.

Instead, I would recommend configuring a timeout that’s slightly greater than the maximum time that you would wait before manually killing the application instance.

But there is no timeout on the ordinary processing of requests, right? If this is the case, then it seems also reasonable to have no timeout on the processing of last requests.

[wilkinsona]

Requests that have the potential to be long-running should be async and will then be subject to the async request timeout. This timeout is in addition to timeouts at the network level.

In a Spring MVC app, a request can be made async by returning DeferredResult, among others, from a controller.

[Azmisov]

When I am testing, I am seeing the container (Tomcat) getting shutdown before the application context / beans have shutdown. From logs, the order is:

1. Shutdown hooks (Runtime.addShutdownHook)
2. Graceful container shutdown (Tomcat)
3. Application context destroyed (ServletContextListener.contextDestroyed)
4. Spring application @PreDestroy

Which is problematic for me: I have some async threads running some computations, and the server is publishing metrics/health of that computation via its API. When shutdown is triggered, I want the computations to gracefully exit, but keep the server running and accepting connections to have metrics/health during that shutdown period. I can wait for the computation to finish in Application context or Predestroy hooks, but the tomcat server shuts down anyways. Any solution that would allow the server to stay up while the async thread is finishing its computation?

[wilkinsona]

You'll have to implement SmartLifecycle with an appropriate phase so that you can participate in the lifecycle. Your implementation can then wait for the async tasks to complete. You'll want it to be called to stop before WebServerStartStopLifecycle is called. This ordering is controlled by the phase. You may also want to consider the phase of WebServerGracefulShutdownLifecycle.

If you have any further questions, please follow up on Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

[Azmisov]

Follow up, for those reading this thread in the future