search

Stratio / cassandra-lucene-index

Lucene based secondary indexes for Cassandra
Apache License 2.0
600 stars 170 forks source link

JVM segfault in ~StubRoutines::updateBytesCRC32 during commit() #415

Closed chipkillmar closed 4 years ago

chipkillmar commented 4 years ago

Thank you for this plugin!

I'm seeing an occasional JVM crash when com.stratio.cassandra.lucene.IndexWriterWide.commit() is called for one of my secondary indices.

This is while running Cassandra 3.0.17 with Oracle JDK 1.8.0_202-b08, although I've noticed that it also happens with 3.0.16.

I'm curious if anyone else has encountered this issue before with the plugin enabled.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa57d000a00, pid=3, tid=0x00007eb3d557d700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_202-b08) (build 1.8.0_202-b08)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.202-b08 mixed mode linux-amd64 )
# Problematic frame:
# v  ~StubRoutines::updateBytesCRC32
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f9d13d5ed00):  JavaThread "SharedPool-Worker-74" daemon [_thread_in_Java, id=1556, stack(0x00007eb3d553d000,0x00007eb3d557e000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f73385f6000

Registers:
RAX=0x0000000056414520, RBX=0x00007fa566bd18d8, RCX=0x00007fa594e3b460, RDX=0x0000000000000131
RSP=0x00007eb3d557bf70, RBP=0x00007eb3d557bf70, RSI=0x00007f73385f5ff0, RDI=0x000000002769c74a
R8 =0x0000000000003ecb, R9 =0x00007f9e8c0005b8, R10=0x00007fa57d000940, R11=0x0000000000003ec1
R12=0x00007eb3d557c770, R13=0x00007fa566bd18c0, R14=0x00007f9d1b9a4ec0, R15=0x00007f9d13d5ed00
RIP=0x00007fa57d000a00, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007eb3d557bf70)
0x00007eb3d557bf70:   00000000003bc466 00007fa5815103d3
0x00007eb3d557bf80:   003bc46666bd1730 00000000003bc466
0x00007eb3d557bf90:   00007fa566bd1730 0000000000ce724d
0x00007eb3d557bfa0:   003c033100003ecb 00007f9f003c0331
0x00007eb3d557bfb0:   00007fa11e085b98 00007fa566bd1730
0x00007eb3d557bfc0:   00007eb3d557bfe0 00007fa5947b8393
0x00007eb3d557bfd0:   000000005dcae29d 4160000000000000
0x00007eb3d557bfe0:   00007f9e8c48037f 0000000000000001
0x00007eb3d557bff0:   00007f9d13d5fce8 0000000000000000
0x00007eb3d557c000:   00007fa566bd1730 00007fa57f8323f0
0x00007eb3d557c010:   00007eb3d557c080 00007fa594307861
0x00007eb3d557c020:   00007f9e8a600680 00007fa57fa3bb88
0x00007eb3d557c030:   0000000000ce724d 00007fa1128712d0
0x00007eb3d557c040:   00007fa566bd16d8 0000000100000001
0x00007eb3d557c050:   0000000000000001 0000000000000000
0x00007eb3d557c060:   0000000000000000 00007fa594776388
0x00007eb3d557c070:   00007f9d13d5f380 00007fa594462f16
0x00007eb3d557c080:   00007fa566bcc030 0000000000000002
0x00007eb3d557c090:   00007fa566bd16d8 00007fa57dc67fd4
0x00007eb3d557c0a0:   0000000000000001 0000000000000003
0x00007eb3d557c0b0:   00007f9e8a6006d8 00007fa566bd16f8
0x00007eb3d557c0c0:   00007f9e8a6006d8 00007fa566bd16f8
0x00007eb3d557c0d0:   00007fa566bd16d8 00007fa594fa796d
0x00007eb3d557c0e0:   00007fa566bd10b0 00007fa57fa26348
0x00007eb3d557c0f0:   00007fa566bd1130 0000000000000003
0x00007eb3d557c100:   0000000000000003 00007fa5800229d8
0x00007eb3d557c110:   00007f9e8b8f4480 00007fa112870fd8
0x00007eb3d557c120:   0000041d00000000 00007f9e8b8f45e8
0x00007eb3d557c130:   00007f9e9dc12978 00007f9f4de4d858
0x00007eb3d557c140:   0000000000000003 00007fa57f92e7a4
0x00007eb3d557c150:   00007fa566bd0fc0 00007fa566bd10b0
0x00007eb3d557c160:   00007fa566bd10b0 00007fa566bd1698 

Instructions: (pc=0x00007fa57d000a00)
0x00007fa57d0009e0:   c4 e3 79 44 e9 11 c4 e3 79 44 c9 00 c5 f1 ef 0e
0x00007fa57d0009f0:   c5 f1 ef cd c4 e3 79 44 ea 11 c4 e3 79 44 d2 00
0x00007fa57d000a00:   c5 e9 ef 56 10 c5 e9 ef d5 c4 e3 79 44 eb 11 c4
0x00007fa57d000a10:   e3 79 44 db 00 c5 e1 ef 5e 20 c5 e1 ef dd c4 e3 

Register to memory mapping:

RAX=0x0000000056414520 is an unknown value
RBX=0x00007fa566bd18d8 is pointing into object: 0x00007fa566876d48

[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007eb3d553d000,0x00007eb3d557e000],  sp=0x00007eb3d557bf70,  free space=251k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
v  ~StubRoutines::updateBytesCRC32
J 31897 C2 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap()V (365 bytes) @ 0x00007fa5815103d3 [0x00007fa58150fe40+0x593]
J 18543 C2 org.apache.cassandra.io.util.SegmentedFile.createReader(J)Lorg/apache/cassandra/io/util/FileDataInput; (12 bytes) @ 0x00007fa57f8323f0 [0x00007fa57f832100+0x2f0]
J 18236 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState.setToBlock(I)V (147 bytes) @ 0x00007fa57fa26348 [0x00007fa57fa26000+0x348]
J 19602 C2 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.slice(Lorg/apache/cassandra/db/Slice;)Ljava/util/Iterator; (66 bytes) @ 0x00007fa57f92e7a4 [0x00007fa57f92e3c0+0x3e4]
J 16731 C2 org.apache.cassandra.db.filter.ClusteringIndexNamesFilter$1.hasNext()Z (123 bytes) @ 0x00007fa57ef4b4d8 [0x00007fa57ef4a8a0+0xc38]
J 33068 C2 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext()Ljava/lang/Object; (5 bytes) @ 0x00007fa58025b6b8 [0x00007fa58025ade0+0x8d8]
J 34056 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;Z)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (814 bytes) @ 0x00007fa5818e6be8 [0x00007fa5818e3fe0+0x2c08]
J 28810 C2 org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(Lorg/apache/cassandra/db/ColumnFamilyStore;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; (23 bytes) @ 0x00007fa580205ce4 [0x00007fa580205b20+0x1c4]
J 33249 C2 com.stratio.cassandra.lucene.IndexWriterWide.commit()V (133 bytes) @ 0x00007fa580fc6b94 [0x00007fa580fc6460+0x734]
J 25195 C2 com.stratio.cassandra.lucene.IndexWriter$$Lambda$245.apply$mcV$sp()V (8 bytes) @ 0x00007fa57d398a98 [0x00007fa57d398a60+0x38]
J 16896 C2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object; (10 bytes) @ 0x00007fa57d5b890c [0x00007fa57d5b88e0+0x2c]
J 14155 C2 org.apache.cassandra.index.SecondaryIndexManager$WriteTimeTransaction.commit()V (34 bytes) @ 0x00007fa57e8a49ec [0x00007fa57e8a4600+0x3ec]
J 33494 C2 org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;Lorg/apache/cassandra/index/transactions/UpdateTransaction;)[J (468 bytes) @ 0x00007fa581810de8 [0x00007fa58180fc20+0x11c8]
J 14859 C2 org.apache.cassandra.db.Memtable.put(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/index/transactions/UpdateTransaction;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;)J (208 bytes) @ 0x00007fa57f276674 [0x00007fa57f275e60+0x814]
J 31222 C2 org.apache.cassandra.db.ColumnFamilyStore.apply(Lorg/apache/cassandra/db/partitions/PartitionUpdate;Lorg/apache/cassandra/index/transactions/UpdateTransaction;Lorg/apache/cassandra/utils/concurrent/OpOrder$Group;Lorg/apache/cassandra/db/commitlog/ReplayPosition;)V (178 bytes) @ 0x00007fa581322394 [0x00007fa581322280+0x114]
J 25267 C2 org.apache.cassandra.db.Keyspace.applyInternal(Lorg/apache/cassandra/db/Mutation;ZZZZLjava/util/concurrent/CompletableFuture;)Ljava/util/concurrent/CompletableFuture; (779 bytes) @ 0x00007fa57e4067f8 [0x00007fa57e405d20+0xad8]
J 25497 C2 org.apache.cassandra.db.MutationVerbHandler.doVerb(Lorg/apache/cassandra/net/MessageIn;I)V (159 bytes) @ 0x00007fa57ee5377c [0x00007fa57ee53440+0x33c]
J 21348 C2 org.apache.cassandra.net.MessageDeliveryTask.run()V (161 bytes) @ 0x00007fa57d866800 [0x00007fa57d866680+0x180]
J 25404% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ 0x00007fa580993c18 [0x00007fa580993000+0xc18]
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68839b]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xddb
V  [libjvm.so+0x685c63]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263
V  [libjvm.so+0x686227]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x6f239c]  thread_entry(JavaThread*, Thread*)+0x6c
V  [libjvm.so+0xa7b9eb]  JavaThread::thread_main_inner()+0xdb
V  [libjvm.so+0xa7bcf1]  JavaThread::run()+0x2d1
V  [libjvm.so+0x90d8c2]  java_start(Thread*)+0x102
C  [libpthread.so.0+0x7e25]  start_thread+0xc5
...
chipkillmar commented 4 years ago

I was able to work around this by disabling memory-mapped file I/O for SSTables in cassandra.yaml:

disk_access_mode: mmap_index_only