Open zhu-he opened 1 year ago
Hi @zhu-he ,
I'm using version 4.5.1 of Actix-web, and when I do kill -SIGTERM {actix server pid}
, I got graceful shutdown logs:
[2024-03-01T15:31:06.361604665Z] INFO: testapp/2721791 on archlinux: starting 32 workers (target=actix_server::builder, line=240)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/builder.rs
[2024-03-01T15:31:06.361654984Z] INFO: testapp/2721791 on archlinux: Actix runtime found; starting in Actix runtime (target=actix_server::server, line=196)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/server.rs
[2024-03-01T15:49:20.526868842Z] INFO: testapp/2721791 on archlinux: SIGTERM received; starting graceful shutdown (target=actix_server::server, line=328)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/server.rs
[2024-03-01T15:49:20.526993052Z] INFO: testapp/2721791 on archlinux: accept thread stopped (target=actix_server::accept, line=143)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/accept.rs
[2024-03-01T15:49:20.527000752Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.526999672Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.526999202Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527008162Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527039163Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527040443Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527045863Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527062153Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527048713Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527065093Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527065673Z] INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
Not sure which version you are using, maybe the maintainers did add this on the later version
This is likely mislabeled if it in fact is already implemented. There's nothing easier than 'nothing'.
When an
actix-web
server receives a SIGTERM signal, it will stop accepting new connections and wait for all active connections to close. After the shutdown timeout configured byHttpServer::shutdown_timeout
, the server will force close all active connections.In our case, we have several servers behind a layer 4 load balancer. When we want to deploy a new version of our service, we will perform a rolling update. We will stop one old server, deploy the new version of the service, and start the new server. Before the old server is stopped, the load balancer will stop making new connections to the old server. However, the old server will still have some persistent connections. If some request is received before the old server is stopped, but not finished before the shutdown timeout, the client will receive a
connection reset by peer
error.After some research, we found that other web frameworks have a more graceful shutdown strategy. For HTTP/1.1, the server will add a
Connection: close
header to all subsequent responses. For HTTP/2, the server will send aGOAWAY
frame to the client. This will allow the client to close the connection gracefully.For example, in the
hyper
crate, the above strategy is implemented in the code. Allhyper
-based web frameworks, such asaxum
, can benefit from this graceful shutdown strategy.We have tried to implement this graceful shutdown strategy in
actix-web
using middleware. However, some difficulties are encountered. For example, the middleware cannot access the shutdown signal, but luckily signals are the only shutdown condition in our case, and we can directly watch them. Another difficulty is that the middleware can only solve the problem in HTTP/1.1, but not in HTTP/2, as the middleware cannot access the underlying connection.We hope that
actix-web
can implement this graceful shutdown strategy. This will makeactix-web
more suitable for production environments.