search

Stratio / cassandra-lucene-index

Lucene based secondary indexes for Cassandra
Apache License 2.0
600 stars 170 forks source link

BufferUnderflowExceptions when deleting table #370

Open smigfu opened 6 years ago

smigfu commented 6 years ago

Hi,

during my unit tests I create a lot of tables with a lucene index applied to it, perform some operations and delete the table afterwards. Quite often but intermittently these test do fail and even impact other tests which don't use the index. This is due to the exception thrown on the server closing active connections on the client side.

Is there a way to more gracefully handle the case in the plugin? I have seen this issue for c* (https://issues.apache.org/jira/browse/CASSANDRA-13868) and tried with a combination of 3.11.1 and a custom build of lucene but the error still persited.

Cheers, Philipp

ERROR [Native-Transport-Requests-3] 2017-12-06 17:09:51,249 QueryMessage.java:129 - Unexpected error during query
java.nio.BufferUnderflowException: null
    at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[na:1.8.0_121]
    at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) ~[na:1.8.0_121]
    at org.apache.cassandra.serializers.CollectionSerializer.readCollectionSize(CollectionSerializer.java:79) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.frozenCollectionSize(ColumnsMapper.scala:243) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:191) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:171) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:119) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$4(ColumnsMapper.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$3(ColumnsMapper.scala:89) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:87) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:56) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.document(IndexPostProcessor.scala:141) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.$anonfun$top$1(IndexPostProcessor.scala:106) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.top(IndexPostProcessor.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.process(IndexPostProcessor.scala:57) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:168) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at org.apache.cassandra.db.PartitionRangeReadCommand.postReconciliationProcessing(PartitionRangeReadCommand.java:330) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:2284) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.PartitionRangeReadCommand.execute(PartitionRangeReadCommand.java:186) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at com.stratio.cassandra.lucene.IndexQueryHandler.executeSortedLuceneQuery(IndexQueryHandler.scala:226) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.executeLuceneQuery(IndexQueryHandler.scala:193) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.processStatement(IndexQueryHandler.scala:122) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.process(IndexQueryHandler.scala:101) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.0.jar:3.11.0]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.0.jar:3.11.0]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
ERROR [Native-Transport-Requests-3] 2017-12-06 17:09:51,268 ErrorMessage.java:384 - Unexpected exception during request
java.nio.BufferUnderflowException: null
    at java.nio.Buffer.nextGetIndex(Buffer.java:506) ~[na:1.8.0_121]
    at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) ~[na:1.8.0_121]
    at org.apache.cassandra.serializers.CollectionSerializer.readCollectionSize(CollectionSerializer.java:79) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.frozenCollectionSize(ColumnsMapper.scala:243) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:191) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:171) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper$.columns(ColumnsMapper.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:119) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$4(ColumnsMapper.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.$anonfun$columns$3(ColumnsMapper.scala:89) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.TraversableOnce.$anonfun$foldRight$1(TraversableOnce.scala:162) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterator.foldRight(Iterator.scala:1409) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractIterable.foldRight(Iterable.scala:54) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.AbstractTraversable.$colon$bslash(Traversable.scala:104) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:87) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.mapping.ColumnsMapper.columns(ColumnsMapper.scala:56) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.document(IndexPostProcessor.scala:141) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.$anonfun$top$1(IndexPostProcessor.scala:106) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:156) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.top(IndexPostProcessor.scala:103) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexPostProcessor.process(IndexPostProcessor.scala:57) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:168) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.ReadCommandPostProcessor.apply(IndexPostProcessor.scala:161) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at org.apache.cassandra.db.PartitionRangeReadCommand.postReconciliationProcessing(PartitionRangeReadCommand.java:330) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:2284) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.PartitionRangeReadCommand.execute(PartitionRangeReadCommand.java:186) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at com.stratio.cassandra.lucene.IndexQueryHandler.executeSortedLuceneQuery(IndexQueryHandler.scala:226) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.executeLuceneQuery(IndexQueryHandler.scala:193) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.processStatement(IndexQueryHandler.scala:122) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at com.stratio.cassandra.lucene.IndexQueryHandler.process(IndexQueryHandler.scala:101) ~[cassandra-lucene-index-plugin-3.11.0.0.jar:na]
    at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.0.jar:3.11.0]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.0.jar:3.11.0]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
smigfu commented 6 years ago

Some more details:

=> all of the above happens intermittently only if I don't add any of the indexed fields to the SELECT clause (in my search case I am only interested in the primary keys which aren't indexed). As soon as I add the indexed fields to my SELECT all of the symptoms above disappear.

That said: Is it a requirement to add the indexed fields to your SELECT? Or do I always need to also add the primary keys to the index?

ealonsodb commented 6 years ago

Hi @smigfu Could you please tell me wich versions(cassandras and index) are you using?. And also, a complete test in CQL?

smigfu commented 6 years ago

I will try to put something together. As said this is an intermittent issue. Would it help if I provide you with the files the index writes to the filesystem?

As for the version of C*: I have seen this with 3.0.10, 3.10.0, 3.11.0 and 3.11.1 (custom build of the plugin)

smiklosovic commented 6 years ago

I am hitting this too.