R2dbcTimeoutException - pool runs out of connections

jivkodobrev commented 8 months ago

We have a high volume processing project using r2dbc-pool and r2dbc-postgresql modules. In some environments we started periodically experiencing issues where attempts to get a connection from pool end up with exception:

org.springframework.dao.DataAccessResourceFailureException, Message: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.spi.R2dbcTimeoutException: Connection acquisition timed out after 120000ms

We captured a couple of heap dumps and observed:

all io.rdbc.pool.PooledConnections have isClosed=true
all io.r2dbc/postgresql.PostgresqlConnection objects they point to seem to be open and idle (client.isClosed = 0, client.transactionStatos = IDLE)
Looking at io.r2dbc.pool.ConnectionPool and its reactor.pool.SimpleDequePool, it has a pendingSize = 7, acquired = 20 (we set the pool size to 20 per container and the idleResources deque seems empty

We suspect that in a period of a spike of error conditions with connections (e.g. lock timeout) there's a racing or timing condition that occurs that prevents connections from being released properly and returned to the pool.

Do you have suggestions for investigating and troubleshooting this issue to it's root cause? Thanks in advance!

SimoneGiusso commented 8 months ago

jivkodobrev commented 8 months ago

@SimoneGiusso Thanks Simone! I've tried the repro from #198 but was not able to reproduce the problem. The issue seems very close to what we are experiencing. I've asked Alexei (the creator of #198) for details.

UPDATE: I was actually able to reproduce the problem after running the app for a while. I am investigating and will keep you posted.

cyberluke commented 5 months ago

@jivkodobrev Hi, do you have any update? Can you share some code example, so I can help you solve it? Thanks

jivkodobrev commented 5 months ago

@cyberluke Hi - unfortunately I got reassigned to something else and will not get the chance to work on this. BTW - check the #198 mentioned above, it seems to be the issue. HTH

r2dbc / r2dbc-pool

R2dbcTimeoutException - pool runs out of connections #206