search

edgedb / edgedb-java

The official Java client library for EdgeDB
Apache License 2.0
19 stars 3 forks source link

query result items count less than should be #15

Closed petersamokhin closed 11 months ago

petersamokhin commented 11 months ago

Hello there! I have a simple query like that:

select foo::Bar.someId

which returns a large set, ±1500 items:

{
  'foo-001',
  'foo-002',
  # many more here
}

If I execute such query in edgedb UI, all good. If I use EdgeDbClient in Java, I get only 600-700 items. I tried to workaround it by using the other methods (queryJson, queryJsonElements) but no luck. The only thing that helped isselect array_join(array_agg(foo::Bar.someId), ',') 🫠

Versions:

Implicit limit is 0 in the config.

petersamokhin commented 11 months ago

It actually looks like it even stopped working lol. I now get some weird errors when I try to select even a large raw string (which is the result of the IDs concatenation for example).

This is definitely an issue in the Java client because the query itself is working, and it also works if you set an explicit small limit. By experimenting it turned out that e.g. 1000 limit still works but 1100 fails

io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: Expected one element array for data, got 31067
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1382)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1245)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1294)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IndexOutOfBoundsException: Expected one element array for data, got 31067
        at com.edgedb.driver.binary.packets.receivable.Data.<init>(Data.java:15)
        at com.edgedb.driver.binary.PacketSerializer.deserialize(PacketSerializer.java:205)
        at com.edgedb.driver.binary.PacketSerializer$1$PacketContract.tryComplete(PacketSerializer.java:144)
        at com.edgedb.driver.binary.PacketSerializer$1.decode(PacketSerializer.java:91)
        at com.edgedb.driver.binary.PacketSerializer$1.decode(PacketSerializer.java:64)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        ... 25 common frames omitted

17:41:27.770 [nioEventLoopGroup-2-1] WARN  c.e.driver.binary.PacketSerializer - Hanging data left inside packet reader of type DATA with length 19071
17:41:27.770 [defaultEventExecutorGroup-3-1] ERROR c.e.d.b.duplexers.ChannelDuplexer - Channel failed
io.netty.handler.codec.DecoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1382)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1245)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1294)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0
        at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454)
        at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1383)
        at io.netty.buffer.AbstractByteBuf.getShort(AbstractByteBuf.java:373)
        at io.netty.buffer.CompositeByteBuf._getShort(CompositeByteBuf.java:968)
        at io.netty.buffer.AbstractByteBuf.getShort(AbstractByteBuf.java:374)
        at io.netty.buffer.CompositeByteBuf._getShort(CompositeByteBuf.java:968)
        at io.netty.buffer.AbstractByteBuf.readShort(AbstractByteBuf.java:750)
        at com.edgedb.driver.binary.PacketReader.readInt16(PacketReader.java:121)
        at com.edgedb.driver.binary.packets.receivable.Data.<init>(Data.java:12)
        at com.edgedb.driver.binary.PacketSerializer.deserialize(PacketSerializer.java:205)
        at com.edgedb.driver.binary.PacketSerializer$1$PacketContract.tryComplete(PacketSerializer.java:144)
        at com.edgedb.driver.binary.PacketSerializer$1.decode(PacketSerializer.java:91)
        at com.edgedb.driver.binary.PacketSerializer$1.decode(PacketSerializer.java:64)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
        ... 25 common frames omitted
quinchs commented 11 months ago

This is most likely an issue with how netty frames data compared to the edgedb protocol format, im investigating now

quinchs commented 11 months ago

Fixed in version 0.2.1, which is now live on central