hyperium / hyper

An HTTP library for Rust
https://hyper.rs
MIT License
14.08k stars 1.55k forks source link

graceful shutdown does not shutdown when i use Google Chrome。 #3576

Closed maojinhua closed 4 months ago

maojinhua commented 4 months ago

Version axum 0.7.4 tower-http 0.5 hyper 1.1.0 googel chrome 121.0.6167.184

Platform macOs Big Sur 11.5.1

Description After clone axum source code, run the following command

cargo run -p example-graceful-shutdown

When the server is up, do the follows in Geegle Chrome:

1.open the serving page http://127.0.0.1:3000/ in browser (this is a must step to reproduce the problem) 2.go back to your terminal and press ctrl+c to stop the server 3.and you see that the server is hanging up but not stop even after a long time 4.After closing Google Chrome, the server shuts down immediately.

this is the log. I found that when Google Chrome initiates a request, it creates two connections. After the request is completed, only one connection is closed, but the other connection remains active。

2024-02-20T16:55:44.219277+08:00  INFO resource_server: listening on 0.0.0.0:8998
2024-02-20T16:55:49.452449+08:00 TRACE axum::serve: connection 127.0.0.1:49501 accepted
2024-02-20T16:55:49.452611+08:00 TRACE axum::serve: connection 127.0.0.1:49502 accepted
2024-02-20T16:55:49.456245+08:00 DEBUG request: tower_http::trace::on_request: started processing request method=GET uri=/slow version=HTTP/1.1
^C2024-02-20T16:55:52.647503+08:00 TRACE axum::serve: received graceful shutdown signal. Telling tasks to shutdown
2024-02-20T16:55:52.647611+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T16:55:52.647609+08:00 TRACE axum::serve: signal received, not accepting new connections
2024-02-20T16:55:52.647668+08:00 TRACE axum::serve: waiting for 2 task(s) to finish
2024-02-20T16:55:52.647629+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T16:55:54.461719+08:00 DEBUG request: tower_http::trace::on_response: finished processing request latency=5005 ms status=200 method=GET uri=/slow version=HTTP/1.1
2024-02-20T16:55:54.462067+08:00 TRACE axum::serve: connection 127.0.0.1:49501 closed
2024-02-20T16:56:50.811452+08:00 TRACE axum::serve: connection 127.0.0.1:49502 closed

The following is the request log sent using Safari browser or Postman. You can see that the server shuts down normally, and there is only one connection initiated for each request sent by the browser.

2024-02-20T17:16:31.602835+08:00  INFO resource_server: listening on 0.0.0.0:8998
2024-02-20T17:16:36.918718+08:00 TRACE axum::serve: connection 127.0.0.1:55361 accepted
2024-02-20T17:16:36.935941+08:00 DEBUG request: tower_http::trace::on_request: started processing request method=GET uri=/slow version=HTTP/1.1
^C2024-02-20T17:16:38.867539+08:00 TRACE axum::serve: received graceful shutdown signal. Telling tasks to shutdown
2024-02-20T17:16:38.867729+08:00 TRACE axum::serve: signal received in task, starting graceful shutdown
2024-02-20T17:16:38.867749+08:00 TRACE axum::serve: signal received, not accepting new connections
2024-02-20T17:16:38.867926+08:00 TRACE axum::serve: waiting for 1 task(s) to finish
2024-02-20T17:16:41.938476+08:00 DEBUG request: tower_http::trace::on_response: finished processing request latency=5002 ms status=200 method=GET uri=/slow version=HTTP/1.1
2024-02-20T17:16:41.938716+08:00 TRACE axum::serve: connection 127.0.0.1:55361 closed

How can I ensure that Google Chrome also doesn't encounter such an issue? It seems like this problem could lead to the server being unable to shut down for extended periods, which appears to be a bug.

maojinhua commented 4 months ago

Geegle Chrome can send request after server shut down. Safari browser or Postman can not send request after server shut down.

mladedav commented 4 months ago

I've tracked this to UpgradeableConnection::graceful_shutdown in hyper-util doing nothing when the connection is still in the UpgradeableConnStateProj::ReadVersion state, i.e. the client hasn't sent the first HTTP request line.

I didn't find anything useful to do there either but maybe the function could signal to the caller that nothing has been read yet so it is safe to just drop the connection?

Alternatively we could take the io inside and close the TCP connection but I fail to see how that can be easily done.


What the browser seems to be doing is it opens a spare connection and leaves it lying around without sending anything on it. This prevents the axum example to finish because it waits for this connection. The example even includes a timeout layer but since the connection never gets to the state where a tower service is called, it's completely ignored.

This can also be simulated with simple nc localhost 3000 and leaving this connection open.

seanmonstar commented 4 months ago

Hm, I thought we fixed that as part 1.0.

mladedav commented 3 months ago

This is ultimately a duplicate of #1885 so that one can be closed now.

Probably, #2294 can be also closed.