Connection usage grows to maximum with even low concurrency

I have a connection pool configured like this:

let pool = r2d2::Pool::builder()
    .min_idle(Some(1))
    .idle_timeout(Some(Duration::from_secs(30)))
    .build(manager)?;

I expected this would approximately match my peak concurrent connection usage (ie. if I never use more than 3 concurrent connections, I would expect my connection usage to sit around 3).

However, in practice I've found that the connection pool starts around the expected size, but then after some time grows to whatever is the upper limit on connections.

I have nine different services deployed to my cluster, all using r2d2, all using the exact same connection pool configuration, and each one with 2-4 replicas. These services are divided into two types: those with background workers (ie. continually polling the database) and those without (pure request/response based).

All of the services with background workers eat up their maximum allocation of connections once they've been running for any non-trivial amount of time. I can also see that all of the connections have been recently active (within the last few seconds), so it's not a question of "leaking" zombie connections somehow.

These services with background workers only use a single thread for polling the database, so despite the fact that they perform constant DB access, they should never use more than 1 concurrent connection when idle.

Based on this, it appears that there's an issue with the way connections are given out from the pool - instead of giving out the "most recently used" connection for reuse, it must be giving out older or random connections from the pool, keeping all of them alive even when not needed.

sfackler / r2d2

Connection usage grows to maximum with even low concurrency #65