Closed chipkillmar closed 4 years ago
Thank you for this plugin!
I'm seeing an occasional JVM crash when com.stratio.cassandra.lucene.IndexWriterWide.commit() is called for one of my secondary indices.
com.stratio.cassandra.lucene.IndexWriterWide.commit()
This is while running Cassandra 3.0.17 with Oracle JDK 1.8.0_202-b08, although I've noticed that it also happens with 3.0.16.
I'm curious if anyone else has encountered this issue before with the plugin enabled.
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fa57d000a00, pid=3, tid=0x00007eb3d557d700 # # JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 ) # Problematic frame: # v ~StubRoutines::updateBytesCRC32 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x00007f9d13d5ed00): JavaThread "SharedPool-Worker-74" daemon [_thread_in_Java, id=1556, stack(0x00007eb3d553d000,0x00007eb3d557e000)] siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f73385f6000 Registers: RAX=0x0000000056414520, RBX=0x00007fa566bd18d8, RCX=0x00007fa594e3b460, RDX=0x0000000000000131 RSP=0x00007eb3d557bf70, RBP=0x00007eb3d557bf70, RSI=0x00007f73385f5ff0, RDI=0x000000002769c74a R8 =0x0000000000003ecb, R9 =0x00007f9e8c0005b8, R10=0x00007fa57d000940, R11=0x0000000000003ec1 R12=0x00007eb3d557c770, R13=0x00007fa566bd18c0, R14=0x00007f9d1b9a4ec0, R15=0x00007f9d13d5ed00 RIP=0x00007fa57d000a00, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e Top of Stack: (sp=0x00007eb3d557bf70) 0x00007eb3d557bf70: 00000000003bc466 00007fa5815103d3 0x00007eb3d557bf80: 003bc46666bd1730 00000000003bc466 0x00007eb3d557bf90: 00007fa566bd1730 0000000000ce724d 0x00007eb3d557bfa0: 003c033100003ecb 00007f9f003c0331 0x00007eb3d557bfb0: 00007fa11e085b98 00007fa566bd1730 0x00007eb3d557bfc0: 00007eb3d557bfe0 00007fa5947b8393 0x00007eb3d557bfd0: 000000005dcae29d 4160000000000000 0x00007eb3d557bfe0: 00007f9e8c48037f 0000000000000001 0x00007eb3d557bff0: 00007f9d13d5fce8 0000000000000000 0x00007eb3d557c000: 00007fa566bd1730 00007fa57f8323f0 0x00007eb3d557c010: 00007eb3d557c080 00007fa594307861 0x00007eb3d557c020: 00007f9e8a600680 00007fa57fa3bb88 0x00007eb3d557c030: 0000000000ce724d 00007fa1128712d0 0x00007eb3d557c040: 00007fa566bd16d8 0000000100000001 0x00007eb3d557c050: 0000000000000001 0000000000000000 0x00007eb3d557c060: 0000000000000000 00007fa594776388 0x00007eb3d557c070: 00007f9d13d5f380 00007fa594462f16 0x00007eb3d557c080: 00007fa566bcc030 0000000000000002 0x00007eb3d557c090: 00007fa566bd16d8 00007fa57dc67fd4 0x00007eb3d557c0a0: 0000000000000001 0000000000000003 0x00007eb3d557c0b0: 00007f9e8a6006d8 00007fa566bd16f8 0x00007eb3d557c0c0: 00007f9e8a6006d8 00007fa566bd16f8 0x00007eb3d557c0d0: 00007fa566bd16d8 00007fa594fa796d 0x00007eb3d557c0e0: 00007fa566bd10b0 00007fa57fa26348 0x00007eb3d557c0f0: 00007fa566bd1130 0000000000000003 0x00007eb3d557c100: 0000000000000003 00007fa5800229d8 0x00007eb3d557c110: 00007f9e8b8f4480 00007fa112870fd8 0x00007eb3d557c120: 0000041d00000000 00007f9e8b8f45e8 0x00007eb3d557c130: 00007f9e9dc12978 00007f9f4de4d858 0x00007eb3d557c140: 0000000000000003 00007fa57f92e7a4 0x00007eb3d557c150: 00007fa566bd0fc0 00007fa566bd10b0 0x00007eb3d557c160: 00007fa566bd10b0 00007fa566bd1698 Instructions: (pc=0x00007fa57d000a00) 0x00007fa57d0009e0: c4 e3 79 44 e9 11 c4 e3 79 44 c9 00 c5 f1 ef 0e 0x00007fa57d0009f0: c5 f1 ef cd c4 e3 79 44 ea 11 c4 e3 79 44 d2 00 0x00007fa57d000a00: c5 e9 ef 56 10 c5 e9 ef d5 c4 e3 79 44 eb 11 c4 0x00007fa57d000a10: e3 79 44 db 00 c5 e1 ef 5e 20 c5 e1 ef dd c4 e3 Register to memory mapping: RAX=0x0000000056414520 is an unknown value RBX=0x00007fa566bd18d8 is pointing into object: 0x00007fa566876d48 [error occurred during error reporting (printing register info), id 0xb] Stack: [0x00007eb3d553d000,0x00007eb3d557e000], sp=0x00007eb3d557bf70, free space=251k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) v ~StubRoutines::updateBytesCRC32 J 31897 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V (365 bytes) @ 0x00007fa5815103d3 [0x00007fa58150fe40+0x593] J 18543 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput; (12 bytes) @ 0x00007fa57f8323f0 [0x00007fa57f832100+0x2f0] J 18236 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState.setToBlock(I)V (147 bytes) @ 0x00007fa57fa26348 [0x00007fa57fa26000+0x348] J 19602 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.slice(Lorg/apache/cassandra/db/Slice;)Ljava/util/Iterator; (66 bytes) @ 0x00007fa57f92e7a4 [0x00007fa57f92e3c0+0x3e4] J 16731 C2 org.apache.cassandra.db.filter.ClusteringIndexNamesFilter$1.hasNext()Z (123 bytes) @ 0x00007fa57ef4b4d8 [0x00007fa57ef4a8a0+0xc38] J 33068 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Ljava/lang/Object; (5 bytes) @ 0x00007fa58025b6b8 [0x00007fa58025ade0+0x8d8] J 34056 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;Z)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (814 bytes) @ 0x00007fa5818e6be8 [0x00007fa5818e3fe0+0x2c08] J 28810 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(Lorg/apache/cassandra/db/ColumnFamilyStore;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (23 bytes) @ 0x00007fa580205ce4 [0x00007fa580205b20+0x1c4] J 33249 C2 com.stratio.cassandra.lucene.IndexWriterWide.commit()V (133 bytes) @ 0x00007fa580fc6b94 [0x00007fa580fc6460+0x734] J 25195 C2 com.stratio.cassandra.lucene.IndexWriter$$Lambda$245.apply$mcV$sp()V (8 bytes) @ 0x00007fa57d398a98 [0x00007fa57d398a60+0x38] J 16896 C2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007fa57d5b890c [0x00007fa57d5b88e0+0x2c] J 14155 C2 org.apache.cassandra.index.SecondaryIndexManager$WriteTimeTransaction.commit()V (34 bytes) @ 0x00007fa57e8a49ec [0x00007fa57e8a4600+0x3ec] J 33494 C2 org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;Lorg/apache/cassandra/index/transactions/UpdateTransaction;)[J (468 bytes) @ 0x00007fa581810de8 [0x00007fa58180fc20+0x11c8] J 14859 C2 org.apache.cassandra.db.Memtable.put(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/index/transactions/UpdateTransaction;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;)J (208 bytes) @ 0x00007fa57f276674 [0x00007fa57f275e60+0x814] J 31222 C2 org.apache.cassandra.db.ColumnFamilyStore.apply(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/index/transactions/UpdateTransaction;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;Lorg/apache/cassandra/db/commitlog/ReplayPosition;)V (178 bytes) @ 0x00007fa581322394 [0x00007fa581322280+0x114] J 25267 C2 org.apache.cassandra.db.Keyspace.applyInternal(Lorg/apache/cassandra/db/Mutation;ZZZZLjava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture; (779 bytes) @ 0x00007fa57e4067f8 [0x00007fa57e405d20+0xad8] J 25497 C2 org.apache.cassandra.db.MutationVerbHandler.doVerb(Lorg/apache/cassandra/net/MessageIn;I)V (159 bytes) @ 0x00007fa57ee5377c [0x00007fa57ee53440+0x33c] J 21348 C2 org.apache.cassandra.net.MessageDeliveryTask.run()V (161 bytes) @ 0x00007fa57d866800 [0x00007fa57d866680+0x180] J 25404% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ 0x00007fa580993c18 [0x00007fa580993000+0xc18] j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x68839b] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xddb V [libjvm.so+0x685c63] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263 V [libjvm.so+0x686227] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47 V [libjvm.so+0x6f239c] thread_entry(JavaThread*, Thread*)+0x6c V [libjvm.so+0xa7b9eb] JavaThread::thread_main_inner()+0xdb V [libjvm.so+0xa7bcf1] JavaThread::run()+0x2d1 V [libjvm.so+0x90d8c2] java_start(Thread*)+0x102 C [libpthread.so.0+0x7e25] start_thread+0xc5 ...
I was able to work around this by disabling memory-mapped file I/O for SSTables in cassandra.yaml:
cassandra.yaml
disk_access_mode: mmap_index_only
Thank you for this plugin!
I'm seeing an occasional JVM crash when
com.stratio.cassandra.lucene.IndexWriterWide.commit()
is called for one of my secondary indices.This is while running Cassandra 3.0.17 with Oracle JDK 1.8.0_202-b08, although I've noticed that it also happens with 3.0.16.
I'm curious if anyone else has encountered this issue before with the plugin enabled.