tkyc
commented
2 days ago This looks like a deadlock. In endRequestInternal() the try-with-resources block calls close() which calls removeOpenStatement(). removeOpenStatement is waiting on the lock, which was acquired earlier by endRequestInternal().
Thanks for the report. Will look into fixing.
Driver version
12.4.2
SQL Server version
Microsoft SQL Server 2022 (RTM-CU15-GDR) (KB5046059) - 16.0.4150.1 (X64) Sep 25 2024 17:34:41
Client Operating System
Ubuntu 20.04.06 LTS
JAVA/JVM version
java version "21.0.2" 2024-01-16 LTS
Table schema
Not needed
Problem description
We have a server that provides the client listeners for a. large number of messaging queue, which means it runs hundreds of threads draining 100+ queues. The thread uses the message to preform a series database inserts/updates.
The problem is that randomly, as the message processing completes so statements and connections are being closed/committed), a thread will get stuck in SQLServrConnection.endRquestInternal(), specifically on the loop in sqlWarnings = originalSqlWarnings; if (null != openStatements) { while (!openStatements.isEmpty()) { try (Statement st = openStatements.get(0)) {} } openStatements.clear(); } Naively, shouldn't this loop not be an infinite loop, but have some form of timeout? Out of hundreds of threads running 24/7, its not unusual for us to lose 5-10 threads every couple of week to being stuck on this loop.
Expected behavior
The code should not get stuck in an infinite loop.
Actual behavior
The thread hangs in an infinite loop.
Error message/stack trace
A sample stack trace is below - note the cpu time.
"AccountListener-1" #305 [2606778] prio=5 os_prio=0 cpu=146605778.46ms elapsed=298559.22s tid=0x00007fdcb427d360 nid=2606778 runnable [0x0000 7fdc9dfdd000] java.lang.Thread.State: RUNNABLE at com.microsoft.sqlserver.jdbc.SQLServerConnection.endRequestInternal(SQLServerConnection.java:7367) at com.microsoft.sqlserver.jdbc.SQLServerConnection43.endRequest(SQLServerConnection43.java:35) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$2.attemptNotifyEndRequest(C3P0PooledConnectionPool.java:182) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.markEndRequest(C3P0PooledConnectionPool.java:360)
Any other details that can be helpful
JDBC trace logs