search

brettwooldridge / HikariCP

光 HikariCP・A solid, high-performance, JDBC connection pool at last.
Apache License 2.0
19.91k stars 2.92k forks source link

Postgres Client thread getting stuck #2204

Closed debraj-manna closed 4 months ago

debraj-manna commented 4 months ago

I am using Hikari with Spring Boot and Postgres with all default settings. I am occasionally observing that my Postgres client thread is getting stuck like below. I have observed for ~ 12 hours and the thread was not doing anything and the was not doing anything. 

"pool-9-thread-1" #352 prio=5 os_prio=0 cpu=28470.75ms elapsed=173935.95s tid=0x00007fd1a0053e50 nid=0x1cb runnable  [0x00007fd17ecfd000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.Net.poll(java.base@17.0.10/Native Method)
    at sun.nio.ch.NioSocketImpl.park(java.base@17.0.10/NioSocketImpl.java:186)
    at sun.nio.ch.NioSocketImpl.park(java.base@17.0.10/NioSocketImpl.java:195)
    at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.10/NioSocketImpl.java:319)
    at sun.nio.ch.NioSocketImpl.read(java.base@17.0.10/NioSocketImpl.java:355)
    at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.10/NioSocketImpl.java:808)
    at java.net.Socket$SocketInputStream.read(java.base@17.0.10/Socket.java:966)
    at sun.security.ssl.SSLSocketInputRecord.read(java.base@17.0.10/SSLSocketInputRecord.java:484)
    at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@17.0.10/SSLSocketInputRecord.java:478)
    at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(java.base@17.0.10/SSLSocketInputRecord.java:70)
    at sun.security.ssl.SSLSocketImpl.readApplicationRecord(java.base@17.0.10/SSLSocketImpl.java:1465)
    at sun.security.ssl.SSLSocketImpl$AppInputStream.read(java.base@17.0.10/SSLSocketImpl.java:1069)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:162)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:129)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:74)
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:467)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2166)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
    at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
    at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:180)
    at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
    at jdk.internal.reflect.GeneratedMethodAccessor244.invoke(Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.10/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.10/Method.java:568)
    at net.ttddyy.dsproxy.proxy.StatementProxyLogic.performQueryExecutionListener(StatementProxyLogic.java:316)
    at net.ttddyy.dsproxy.proxy.StatementProxyLogic.access$700(StatementProxyLogic.java:37)
    at net.ttddyy.dsproxy.proxy.StatementProxyLogic$1.execute(StatementProxyLogic.java:123)
    at net.ttddyy.dsproxy.listener.MethodExecutionListenerUtils.invoke(MethodExecutionListenerUtils.java:42)
    at net.ttddyy.dsproxy.proxy.StatementProxyLogic.invoke(StatementProxyLogic.java:120)
    at net.ttddyy.dsproxy.proxy.jdk.PreparedStatementInvocationHandler.invoke(PreparedStatementInvocationHandler.java:37)
    at jdk.proxy2.$Proxy144.execute(jdk.proxy2/Unknown Source)
    at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:219)
    at org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:4594)
    at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:236)
    at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:330)
    at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301)
    at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322)
    at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861)
    at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509)
    at org.jooq.impl.ResultQueryTrait.fetchOneInto(ResultQueryTrait.java:533)
    at org.jooq.impl.ResultQueryTrait.fetchOptionalInto(ResultQueryTrait.java:716)
    at com.sp.gossiper.producers.GossiperProducerScheduler.readCheckpoint(GossiperProducerScheduler.java:369)
        ...

Can someone let me know if there are some known issue with the Hikari version I am using or there are certain timeout settings in Hikari which will not allow me to get into this situation?

debraj-manna commented 4 months ago

It looks like I am running into the issue as mentioned in https://github.com/brettwooldridge/HikariCP/wiki/Rapid-Recovery.

I am setting socketTimeout to 30 seconds.

val ds = new HikariDataSource();
ds.addDataSourceProperty("socketTimeout", 30);