Server: detect and shutdown idle connections

[ramiroaisen]

Hi there.

We are using hyper and hyper_util to build a custom reverse proxy to replace nginx from our infrastructure.

We are very happy with it and it's working very nice for us.

The issue we are facing in our production servers is that we have connections that keep idle for large amounts of time (if not forever).

Our code looks something like this:

let graceful = hyper_util::server::graceful::GracefulShutdown::new();
let http = hyper_util::server::conn::auto::Builder::new();

// on tls accept and handshake
let io = TokioIo::new(tls_stream);
let conn = http.serve_connection_with_upgrades(io, service).into_owned();
let watched = graceful.watch(conn);

tokio::spawn(async move {
  if let Err(e) = watched.await {
    log::warn!("error handling connection tls - {e}: {e:?}");
  }
});

But the problem is that many tasks that are spawned this way will never resolve, and be in idle state potentially forever.

Seeing this in tokio-console shows that this tasks keep in idle state for hours and are never polled after the first few polls.

This accumulates in the system, as we are seeing an increment in memory consumption over time.

We tried both adding http2 keep alive to the conn builder, and tcp keepalive to the sockets but that didn't solved the issue.

Is there a way we can shutdown connections that are idle for a given time?

Thank you very much!

system

Alma Linux 9.4 
x86_64 libc 2.34
kernel 5.14.0

versions

hyper = "1.3.1"
hyper_util = "0.1.5"
hyper_rustls = "0.27.1"
tokio = "1.37.0"

[seanmonstar]

Most of the discussion is likely in #1628.

[ramiroaisen]

Thanks for the fast reply.

We implemented a Read/Write timeout on the socket level for both http1 and http2 in plain-text and ssl modes.

I leave here the code if other people want to use it.

https://gist.github.com/ramiroaisen/4bf946a3906b4aa6b090bdab2a4c6688

I'll close this now, thank you very much.