programatik29 / axum-server

High level server designed to be used with axum framework.
MIT License
166 stars 54 forks source link

fix: propagate graceful shutdown to inner hyper connection #119

Open phlip9 opened 1 month ago

phlip9 commented 1 month ago

The hyper v1 migration PR (#93) accidentally (?) removed the call to serve_future.graceful_shutdown(), which prevents the server from shutting down until all client connections close on their own (or we hit the hard shutdown timeout).

In practice, this meant we always hit the hard shutdown timeout, since most clients use connection pools with connection keepalive.

With this change, graceful shutdown will now correctly stop handling any new requests after it's triggered, while letting existing requests complete (within the timeout of course).

The fix itself is just one line:

                        let serve_future = builder.serve_connection_with_upgrades(io, service);
                        tokio::pin!(serve_future);

                        tokio::select! {
                            biased;
                            _ = watcher.wait_graceful_shutdown() => {
+                               serve_future.as_mut().graceful_shutdown();
                                tokio::select! {
                                    biased;
                                    _ = watcher.wait_shutdown() => (),
                                    _ = &mut serve_future => (),
                                }
                            }
                            _ = watcher.wait_shutdown() => (),
                            _ = &mut serve_future => (),
                        }

The bulk of the PR is fixing the graceful shutdown tests.

Fixes #114

phlip9 commented 1 month ago

cc @abs0luty @programatik29

Benjamin-L commented 1 month ago

We were having issues with graceful shutdown delays in grapevine, and have confirmed that this patch fixes them. Shutdown now happens in ~30ms.

phlip9 commented 5 days ago

cc @abs0luty @programatik29

Would be great to get this merged upstream : )