search

userver-framework / userver

Production-ready C++ Asynchronous Framework with rich functionality
https://userver.tech
Apache License 2.0
2.41k stars 282 forks source link

Graceful shutdown #713

Open Anton3 opened 3 weeks ago

Anton3 commented 3 weeks ago

As of now, upon receiving SIGTERM, userver returns 503 to all existing requests, breaks all connections and cancels all handler tasks immediately.

It is advised to set up some cooperation between balancers and hand-written service shutdown scripts to first signal balancers to move traffic to other hosts, then after N seconds send SIGTERM to the userver-based service.

It's suggested that graceful shutdown is optionally built into userver itself. How it should work:

  1. The service receives SIGTERM
  2. handler-ping starts to give out 503 or 500
  3. A certain number of seconds passes
    • Can be specified e.g. in the config of server or handler-ping
    • Hopefully, the health check probe will fail at least once during that time, notifying the balancer
  4. The service starts to actually shut down
Anton3 commented 3 weeks ago

An additional feature request [might be lower priority] is to allow current requests to go past "soft deadline" without cancellation, until they hit "hard deadline", then 503 should be returned.

Example: a long Clickhouse DB query, which can take multiple seconds to complete, and it will hurt if it's cancelled, because there are no transactions. If no long queries are being processed, then the service will shut down immediately upon hitting the soft deadline. If there is a long query in processing, then the service will take its time if needed.