search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.62k stars 3.26k forks source link

[Bug] JDBC OutOfMemoryError: Java heap space #16538

Open gojumprope opened 1 year ago

gojumprope commented 1 year ago

Search before asking

Version

1.2.1

What's Wrong?

04:28:04.628 [Thread-60] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1093 - Exception during pool initialization. org.postgresql.util.PSQLException: This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:907) ~[5564287785892149047.378f8a2ddab2564a281e5f852800e2e9.jar:42.5.1] at org.postgresql.jdbc.PgConnection.setTransactionIsolation(PgConnection.java:966) ~[5564287785892149047.378f8a2ddab2564a281e5f852800e2e9.jar:42.5.1] at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:413) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:363) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.init(JdbcExecutor.java:229) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.(JdbcExecutor.java:66) ~[java-udf-jar-with-dependencies.jar:?] Exception in thread "mssql-jdbc-shared-timer-core-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)

10:52:21.740 [Thread-28] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-3030 - Exception during pool initialization. java.lang.IllegalArgumentException: Self-suppression not permitted at java.lang.Throwable.addSuppressed(Throwable.java:1072) ~[?:1.8.0_352] at com.microsoft.sqlserver.jdbc.SQLServerConnection.isValid(SQLServerConnection.java:6863) ~[7012254909613997775.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.zaxxer.hikari.pool.PoolBase.checkValidationSupport(PoolBase.java:458) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.checkDriverSupport(PoolBase.java:441) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.setupConnection(PoolBase.java:410) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:363) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.init(JdbcExecutor.java:229) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.(JdbcExecutor.java:66) ~[java-udf-jar-with-dependencies.jar:?] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

04:37:04.141 [Thread-53] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-2606 - Exception during pool initialization. com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing". at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3806) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1907) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:3329) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2950) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:2790) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1663) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1064) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:115) ~[java-udf-jar-with-dependencies.jar:?] at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.init(JdbcExecutor.java:229) ~[java-udf-jar-with-dependencies.jar:?] at org.apache.doris.udf.JdbcExecutor.(JdbcExecutor.java:66) ~[java-udf-jar-with-dependencies.jar:?] Caused by: javax.net.ssl.SSLException: Unexpected rethrowing at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:1.8.0_352] at sun.security.ssl.TransportContext.fatal(TransportContext.java:296) ~[?:1.8.0_352] at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:1.8.0_352] at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:1.8.0_352] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) ~[?:1.8.0_352] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1796) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] ... 14 more Caused by: java.io.IOException: Connection reset ClientConnectionId:385b5ae0-fe66-48fc-abd3-3bf14368fc05 at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.ensureSSLPayload(IOBuffer.java:909) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.readInternal(IOBuffer.java:969) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.read(IOBuffer.java:962) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.readInternal(IOBuffer.java:1208) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.read(IOBuffer.java:1195) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) ~[?:1.8.0_352] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305) ~[?:1.8.0_352] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) ~[?:1.8.0_352] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1796) ~[1414212399189005683.5d53b0cb64ec2e5268a5f7a349889d35.jar:?] ... 14 more

What You Expected?

should not hit out of memory error if there is still memory left in the server

How to Reproduce?

SELECT * FROM external jdbc table

query the same table and it become slower and slower but after restart backend, everything back to normal

First Time Execution: 3.183s

Second Time Execution: 3.928s

Third Time Execution: 4.248s

...

N Time Execution: 16.362s

...

java.lang.OutOfMemoryError: GC overhead limit exceeded

After that, doris cannot query anything even backends are still alive, need to restart it only can work

Anything Else?

image

Are you willing to submit PR?

Code of Conduct

Raven888888 commented 1 year ago

Bump, same issue here

Raven888888 commented 10 months ago

Any update?