eclipse-vertx / vertx-sql-client

High performance reactive SQL Client written in Java
Apache License 2.0
894 stars 200 forks source link

DB2 query throws IndexOutOfBoundsException #1033

Open gbadner opened 3 years ago

gbadner commented 3 years ago

Version

4.1.3

Context

Using DB2, the following query:

select table_name as TABLE_NAME, column_name as COLUMN_NAME,  
             case when data_type = 'CHARACTER VARYING' then 'VARCHAR' else data_type end as TYPE_NAME, 
             null as COLUMN_SIZE, null as DECIMAL_DIGITS, is_nullable as IS_NULLABLE, null as DATA_TYPE 
from sysibm.columns 
where true  and table_catalog = ? 
order by table_catalog, table_schema, table_name, column_name, ordinal_position

throws:

 java.lang.IndexOutOfBoundsException: readerIndex(32741) + length(17) exceeds writerIndex(32757): ReadOnlyByteBuf(ridx: 32741, widx: 32757, cap: 32757/32757, unwrapped: PooledSlicedByteBuf(ridx: 0, widx: 32757, cap: 32757/32757, unwrapped: PooledUnsafeDirectByteBuf(ridx: 32860, widx: 32860, cap: 65536)))

See below for full stacktrace.

Do you have a reproducer?

No, but it should be easy to reproduce by adding the query to an existing test, since it only relies on sysibm.columns (DB2's version of information_schema).

We can provide a reproducer using Hibernate ORM and Reactive PRs. Please let me know if you'd like us to do that.

Steps to reproduce

  1. ...
  2. ...
  3. ...
  4. ...

Extra

Full stacktrace:

java.util.concurrent.CompletionException: java.lang.IndexOutOfBoundsException: readerIndex(32741) + length(17) exceeds writerIndex(32757): ReadOnlyByteBuf(ridx: 32741, widx: 32757, cap: 32757/32757, unwrapped: PooledSlicedByteBuf(ridx: 0, widx: 32757, cap: 32757/32757, unwrapped: PooledUnsafeDirectByteBuf(ridx: 32860, widx: 32860, cap: 65536)))
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_302]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_302]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607) ~[?:1.8.0_302]
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[?:1.8.0_302]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_302]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_302]
        at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.sqlclient.impl.QueryResultBuilder.tryFail(QueryResultBuilder.java:118) ~[vertx-sql-client-4.1.3.jar:4.1.3]
        at io.vertx.core.Promise.fail(Promise.java:89) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.Promise.handle(Promise.java:53) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.Promise.handle(Promise.java:29) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.onFailure(PromiseImpl.java:54) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl$ListenerArray.onFailure(FutureImpl.java:268) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.onFailure(PromiseImpl.java:54) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:43) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.sqlclient.impl.command.CommandResponse.fire(CommandResponse.java:46) ~[vertx-sql-client-4.1.3.jar:4.1.3]
        at io.vertx.sqlclient.impl.SocketConnectionBase.handleMessage(SocketConnectionBase.java:279) ~[vertx-sql-client-4.1.3.jar:4.1.3]
        at io.vertx.sqlclient.impl.SocketConnectionBase.lambda$init$0(SocketConnectionBase.java:98) ~[vertx-sql-client-4.1.3.jar:4.1.3]
        at io.vertx.core.net.impl.NetSocketImpl.lambda$new$1(NetSocketImpl.java:97) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.net.impl.NetSocketImpl.lambda$handleMessage$9(NetSocketImpl.java:390) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:274) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:22) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:389) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:154) ~[vertx-core-4.1.3.jar:4.1.3]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.vertx.db2client.impl.codec.DB2Encoder.lambda$write$0(DB2Encoder.java:72) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.DB2Decoder.decodePayload(DB2Decoder.java:96) [vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.DB2Decoder.decode(DB2Decoder.java:52) [vertx-db2-client-4.1.3.jar:4.1.3]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.67.Final.jar:4.1.67.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: java.lang.IndexOutOfBoundsException: readerIndex(32741) + length(17) exceeds writerIndex(32757): ReadOnlyByteBuf(ridx: 32741, widx: 32757, cap: 32757/32757, unwrapped: PooledSlicedByteBuf(ridx: 0, widx: 32757, cap: 32757/32757, unwrapped: PooledUnsafeDirectByteBuf(ridx: 32860, widx: 32860, cap: 65536)))
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) ~[netty-buffer-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1428) ~[netty-buffer-4.1.67.Final.jar:4.1.67.Final]
        at io.netty.buffer.AbstractByteBuf.skipBytes(AbstractByteBuf.java:971) ~[netty-buffer-4.1.67.Final.jar:4.1.67.Final]
        at io.vertx.db2client.impl.drda.Cursor.skipFdocaBytes(Cursor.java:1669) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.drda.Cursor.calculateColumnOffsetsForRow_(Cursor.java:1488) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.drda.Cursor.stepNext(Cursor.java:220) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.drda.Cursor.next(Cursor.java:232) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.RowResultDecoder.next(RowResultDecoder.java:52) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.ExtendedQueryCommandBaseCodec.decodePreparedQuery(ExtendedQueryCommandBaseCodec.java:81) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.ExtendedQueryCommandCodec.decodeQuery(ExtendedQueryCommandCodec.java:49) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.QueryCommandBaseCodec.decodePayload(QueryCommandBaseCodec.java:69) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        at io.vertx.db2client.impl.codec.DB2Decoder.decodePayload(DB2Decoder.java:79) ~[vertx-db2-client-4.1.3.jar:4.1.3]
        ... 21 more
DavideD commented 3 years ago

I've created a test on this branch: https://github.com/DavideD/vertx-sql-client/commit/d117691d3486f2ef2c4d7ec26670148a57b9c3a3

DavideD commented 3 years ago

Any update on this issue?

vietj commented 3 years ago

@mswatosh ?

vietj commented 3 years ago

can you provide a reproducer for SQL client ?

DavideD commented 3 years ago

I've created a test for QueryVariationsTest:

  @Test
  public void testSelectSysibmColumns(TestContext ctx) {
    connect(ctx.asyncAssertSuccess(conn -> conn
        .query("select table_name as TABLE_NAME, column_name as COLUMN_NAME, " +
                   "             case when data_type = 'CHARACTER VARYING' then 'VARCHAR' else data_type end as TYPE_NAME, " +
                   "             null as COLUMN_SIZE, is_nullable as IS_NULLABLE " +
                   "from sysibm.columns " +
                   "order by table_catalog, table_schema, table_name, column_name, ordinal_position" )
        .execute(
          ctx.asyncAssertSuccess(rowSet -> {
            ctx.assertEquals(1, rowSet.size());
            ctx.assertEquals(Arrays.asList("TABLE_NAME", "COLUMN_NAME", "COLUMN_SIZE", "IS_NULLABLE"), rowSet.columnsNames());
            RowIterator<Row> rows = rowSet.iterator();
            ctx.assertTrue(rows.hasNext());
            Row row = rows.next();
            // TODO: Add assertions about the content of the row
            ctx.assertFalse(rows.hasNext());
            conn.close();
          })
        )
    ));
  }

You should be able to cherry-pick this commit: https://github.com/DavideD/vertx-sql-client/commit/d117691d3486f2ef2c4d7ec26670148a57b9c3a3 (It's been a while)

vietj commented 3 years ago

thanks a lot @DavideD I'll have a look

mswatosh commented 3 years ago

@vietj I won't have time until monday to dig into this one, but let me know if you'd like me to take a look then. I took a quick scan but I didn't see an obvious quick fix.

vietj commented 3 years ago

thanks, I'll have a look today as there is a reproducer, if I don't find I'll get back to you :-)

On Thu, Oct 21, 2021 at 9:57 PM Mark Swatosh @.***> wrote:

@vietj https://github.com/vietj I won't have time until monday to dig into this one, but let me know if you'd like me to take a look then. I took a quick scan but I didn't see an obvious quick fix.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/eclipse-vertx/vertx-sql-client/issues/1033#issuecomment-948956555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXDCVIHHGA6WVW7LIXIVDUIBWA7ANCNFSM5DHLSAIQ .

vietj commented 3 years ago

@mswatosh this seems to go beyond my knowledge of DRDA, can you handle it ?

mswatosh commented 2 years ago

It looks like the issue here is that the response exceeds maximum length for a DSS response, so the driver needs to send a CNTQRY to get more data. Unfortunately, it looks like most of the path for that is not implemented, including having the cursor handle the partial row.

vietj commented 2 years ago

is that something you can contribute @mswatosh ?

mswatosh commented 2 years ago

@vietj I should be able to, it's just looking more like a small feature than the quick bug fix I was hoping for.

vietj commented 2 years ago

is there anything you can do about it @mswatosh ?

mswatosh commented 2 years ago

@vietj I've made some progress in my local environment, but I haven't had a lot of time and it's slow going since I have to learn how the code works as I go.