rbygrave
commented
2 months ago So this is all about Graceful shutdown of an application.
In my terms Graceful shutdown needs to run in order of:
- Stop TCP listener, no longer accept new requests
- Quiesce waiting for active requests to complete
- Gracefully stop any Queue message consumers if there are any
- Gracefully stop any Queue message producers if there are any
- Gracefully stop any non-ebean background threads
- *[ebean] Gracefully stop background threads
- *[ebean] Shutdown connection pools
What this means wrt Ebean is:
- Ebean should NOT register it's shutdown hook - ShutdownManager.deregisterShutdownHook()
- Application needs to control the order of shutdown as per above
- The last 2 can be done via ShutdownManager.shutdown()
When application receive SIGTERM, ThreadPoolExecutor used by Ebean to get futureCount should wait that all current HTTP request have been processed before shutting down.
Yes. IMO there is more details to it but yes the application must control the order in which resources are shutdown/closed. Good Dependency Injection libraries give you control over shutdown order / prioritise the order of execution of PreDestroy methods.
Are there any followup questions?
Edit: fwiw, in terms of "Quiesce waiting for active requests to complete" different http servers have different costs associated with doing this like maintaining an AtomicLong counter for active requests (so there is some concurrency cost associated with that). Helidon SE 4.x has a zero runtime cost way to do this so which is great imo - I contributed to get that because I really desire fast graceful shutdown, it's a high priority for me. Some other http servers (like Jetty) have extra add-on features that you need to enable to get this. Some http servers tend to just "wait some amount of time" which imo is not great if you want fast graceful shutdown (and I tend to want that for K8s - speed of shutdown translates directly to speed of blue/green deployment etc).
AntoineDuComptoirDesPharmacies
To be honest, I do not know if this issue should be created here or on the following project : https://github.com/playframework/play-ebean
Context
We are running Play Framework App with Ebean embedded on AWS Fargate using a mix of SPOT provider and standard provider to reach HA. This morning, as often, we had a preemption of our SPOT instance, so receiving a SIGTERM on our app. This is a normal behavior as this node will then relaunch using standard provider.
Expected behavior
When application receive SIGTERM, ThreadPoolExecutor used by Ebean to get
futureCountshould wait that all current HTTP request have been processed before shutting down.Actual behavior
When application receive SIGTERM, ThreadPoolExecutor used by Ebean to get
futureCountshutdown immediately, so that if a request is being processed in the same lapse of time, it will be rejected by ThreadPoolExecutor.Steps to reproduce
pekko.coordinated-shutdown.phases.service-unbind.timeout = 35
Yours faithfully, LCDP