brettwooldridge / HikariCP

光 HikariCP・A solid, high-performance, JDBC connection pool at last.
Apache License 2.0
20.02k stars 2.94k forks source link

Intermittent, non deterministic ArrayIndexOutOfBoundsException JDBC Array Out of Bounds exception on some queries #2066

Open ahaq0 opened 1 year ago

ahaq0 commented 1 year ago

Hi All,

I'm dealing with this intermittent ArrayIndexOutOfBoundsException ocurring when attempting to run certain INSERT queries that work fine sometimes and fail otherwise.

Setup Hikari 5.0.1 my-sql-connector-java 8.0.31

my-sql 8.0.30

Here is the stack trace:

java.sql.SQLException: null at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) 
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:763) 
at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:648) 
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) 
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) 
at io.vertx.ext.jdbc.impl.actions.JDBCExecute.execute(JDBCExecute.java:44) 
at io.vertx.ext.jdbc.impl.actions.JDBCExecute.execute(JDBCExecute.java:29) 
at io.vertx.ext.jdbc.impl.JDBCConnectionImpl.lambda$schedule$3(JDBCConnectionImpl.java:219) 
at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) 
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) 
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) 
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.ArrayIndexOutOfBoundsException: null 
at com.mysql.cj.protocol.a.NativePacketPayload.readInteger(NativePacketPayload.java:398) 
at com.mysql.cj.protocol.a.NativePacketPayload.readString(NativePacketPayload.java:605) 
at com.mysql.cj.protocol.a.NativeServerSessionStateController$NativeServerSessionStateChanges.init(NativeServerSessionStateController.java:112) 
at com.mysql.cj.protocol.a.result.OkPacket.parse(OkPacket.java:66) 
at com.mysql.cj.protocol.a.NativeProtocol.readServerStatusForResultSets(NativeProtocol.java:1755) 
at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:116) 
at
com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:48) 
at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1664) at com.mysql.cj.protocol.a.NativeProtocol.readAllResults(NativeProtocol.java:1718) 
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1064) 
at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:997) 
at com.mysql.cj.NativeSession.execSQL(NativeSession.java:663) 
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:723) ... 14 common frames omitted`

the issue appears reminisicent of the previous bug here https://github.com/brettwooldridge/HikariCP/issues/1712 however, we are now 8.0.31 of my-sql-connector java and the issue seems to occur on 1 of 4 environments with consistent config randomly.

For instance, previously, the query would fail 9 / 10 times across all pods and started occuring out of nowhere i.e. there was no update or config changes that spurred it.

Later, we were able to stop the error by truncating the associated the table. This has appeared to work so far.

Upon digging we found that potentially there is an issue of inserting a double in a Decimal 10,0 column. Was about to fix that but if it was the culprit why would it stop occuring after truncating the table? Further, why would this error occur in only one environment ( an environment that gets more traffic)

One thing to note the environment where this issue is occuring has ongoing activity ( load tests) could load ( and the associated records in the table) be a factor?

Any insights would be sincerely appreciated!

Thanks

ahaq0 commented 1 year ago

Just to add more information, found out that this issue occurs when inserting a record with id greater than 2^16

Also, to make matters more interesting, the issue can be mitigated by making sure that the insert queries fit the correct decimal format.

Can provide more information as necessary

himeshmyk commented 1 year ago

Not sure if this will work for others; we were also getting this error when we upgraded our application to Spring Boot 2.5.x

Nothing seemed to work to fix the ArrayIndexOutOfBoundsException We were also getting this on random tables in different applications.

We came across this 1 thread here - https://bugs.mysql.com/bug.php?id=105706 After adding the mariadb-java-client dependency and switching to the org.mariadb.jdbc.Driver driver class name according to the steps here - https://springframework.guru/configuring-spring-boot-for-mariadb/ Errors seem to have stopped in last 1 day. (We're still monitoring these)

FYI: Our DBs are MariaDBs, maybe that's why this worked for us.