Chronicle map crashed JVM with SigSegv

[bhushankh]

We are using the chronicle map with large dataset. It is consistently crashing the JVM in one of the environments with a SIGSEGV. Stack below. This is a Redhat with OpenJDK build 1.8.0_232-b09.

--------------- T H R E A D ---------------

Current thread (0x00007f3a282f8000): JavaThread "BaseServer_WorkingThread-8" [_thread_in_vm, id=23436, stack(0x00007f399fb04000,0x00007f399fc05000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007f399be93e00

Registers: RAX=0x00007f399bf93008, RBX=0x0000000000100000, RCX=0x00007f3968180210, RDX=0xfffffffffffe01c0 RSP=0x00007f399fc03408, RBP=0x00007f399fc03450, RSI=0x4553038103ce4353, RDI=0x00007f399be93010 R8 =0x0000000000000000, R9 =0x00007f3a282f8000, R10=0x00007f4960aabb67, R11=0x00007f4960aabb18 R12=0x0000000000000000, R13=0x00007f4975da59a0, R14=0x00007f3a282f8000, R15=0x00007f3968080218 RIP=0x00007f497557ca84, EFLAGS=0x0000000000010296, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f399fc03408) 0x00007f399fc03408: 00007f4975874d74 0000000000000000 0x00007f399fc03418: 00007f399be93010 0000000000000170 0x00007f399fc03428: 00007f3acc138660 0000000000000170 0x00007f399fc03438: 00007f399fc034c0 00007f399fc035a8 0x00007f399fc03448: 00007f3a282f8000 00007f399fc034a0 0x00007f399fc03458: 00007f4960aabbe1 0000000000100000 0x00007f399fc03468: 00007f3b3c9b1a60 0000000000000000 0x00007f399fc03478: 00007f3a282f8000 0000000000000000 0x00007f399fc03488: 00007f4975812459 0000000000000000 0x00007f399fc03498: 00007f399fc03500 00007f399fc03548 0x00007f399fc034a8: 00007f49607b7ad4 000000000000000b 0x00007f399fc034b8: 00007f49607b7ad4 0000000000100000 0x00007f399fc034c8: 00007f497549bad6 00007f3968080218 0x00007f399fc034d8: 00007f49607b75f0 0000000000000000 0x00007f399fc034e8: 00007f399be93010 00007f399fc03540 0x00007f399fc034f8: 0000000000000000 00007f3b3c9b1a60 0x00007f399fc03508: 00007f399fc03508 00007f399f943ad4 0x00007f399fc03518: 00007f399fc035a8 00007f399f94a058 0x00007f399fc03528: 00007f399e1e1118 00007f399f943b68 0x00007f399fc03538: 00007f399fc034c0 00007f399fc035b0 0x00007f399fc03548: 0000000002e42922 00007f49642c79c4 0x00007f399fc03558: 0000000000100000 0000000000000000 0x00007f399fc03568: 0000000002242922 00007f399fc035a0 0x00007f399fc03578: 00007f3968080218 000000005e46c928 0x00007f399fc03588: 0000000000000000 00007f399be93010 0x00007f399fc03598: 6485336240bf2500 0000000000000000 0x00007f399fc035a8: 00007f3b4a799a50 00007f3b2abdb920 0x00007f399fc035b8: 0000000000000000 00007f3b5a413c18 0x00007f399fc035c8: 00007f3b00004782 00007f3b6784c2a8 0x00007f399fc035d8: 00000000455b9218 00007f399b293010 0x00007f399fc035e8: 00007f3b2fb3ffe8 0000000002e42922 0x00007f399fc035f8: 000000000000991f 0000000000000170

Instructions: (pc=0x00007f497557ca84) 0x00007f497557ca64: 74 d1 08 48 83 c2 01 75 f0 c3 66 90 48 8b 74 d0 0x00007f497557ca74: e8 48 89 74 d1 e8 48 8b 74 d0 f0 48 89 74 d1 f0 0x00007f497557ca84: 48 8b 74 d0 f8 48 89 74 d1 f8 48 8b 34 d0 48 89 0x00007f497557ca94: 34 d1 48 83 c2 04 7e d4 48 83 ea 04 7c bb c3 48

Register to memory mapping:

RAX=0x00007f399bf93008 is an unknown value RBX=0x0000000000100000 is an unknown value RCX=0x00007f3968180210 is an unknown value RDX=0xfffffffffffe01c0 is an unknown value RSP=0x00007f399fc03408 is pointing into the stack for thread: 0x00007f3a282f8000 RBP=0x00007f399fc03450 is pointing into the stack for thread: 0x00007f3a282f8000 RSI=0x4553038103ce4353 is an unknown value RDI=0x00007f399be93010 is an unknown value R8 =0x0000000000000000 is an unknown value R9 =0x00007f3a282f8000 is a thread R10=0x00007f4960aabb67 is at entry_point+103 in (nmethod)0x00007f4960aab990 R11=0x00007f4960aabb18 is at entry_point+24 in (nmethod)0x00007f4960aab990 R12=0x0000000000000000 is an unknown value R13=0x00007f4975da59a0: <offset 0xfce9a0> in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so at 0x00007f4974dd7000 R14=0x00007f3a282f8000 is a thread R15=0x00007f3968080218 is an unknown value

Stack: [0x00007f399fb04000,0x00007f399fc05000], sp=0x00007f399fc03408, free space=1021k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x7a5a84] J 6283 sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V (0 bytes) @ 0x00007f4960aabbe1 [0x00007f4960aabb00+0xe1] j net.openhft.chronicle.core.UnsafeMemory.copyMemory0(Ljava/lang/Object;JLjava/lang/Object;JJ)V+28 J 25380 C2 net.openhft.chronicle.queue.ExcerptAppender.writeBytes(Lnet/openhft/chronicle/bytes/Bytes;)V (8 bytes) @ 0x00007f49642c79c4 [0x00007f49642c6e80+0xb44] J 23986 C2 com.webaction.runtime.BufferManagement.BufferManagerWithChronicleQueue.put(Lcom/webaction/proc/events/WAEvent;)V (312 bytes) @ 0x00007f496373326c [0x00007f4963731ba0+0x16cc] J 23891 C2 com.webaction.proc.BufferBasedTxnDetails.addEvent(Ljava/lang/Comparable;Lcom/webaction/proc/events/WAEvent;)Ljava/lang/Long; (53 bytes) @ 0x00007f4963de7610 [0x00007f4963de75a0+0x70] J 25709 C2 com.webaction.proc.TxnCache.update(Lcom/webaction/proc/events/WAEvent;)V (1595 bytes) @ 0x00007f496453a138 [0x00007f4964537b20+0x2618] J 22850 C2 com.webaction.proc.OracleReader_1_0.receiveImpl(ILcom/webaction/event/Event;)V (762 bytes) @ 0x00007f4962e6574c [0x00007f4962e655e0+0x16c] J 23700% C2 com.webaction.runtime.components.Source.run()V (203 bytes) @ 0x00007f4963ca7298 [0x00007f4963ca7120+0x178] J 6877 C1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007f49614a988c [0x00007f49614a9780+0x10c] J 17246 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007f4962d5e0c4 [0x00007f4962d5de80+0x244] j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [libjvm.so+0x6802ee] V [libjvm.so+0x67d6b4] V [libjvm.so+0x67dcc9] V [libjvm.so+0x6c33d1] V [libjvm.so+0xa77ac2] V [libjvm.so+0x8c3eb2] C [libpthread.so.0+0x7ea5] start_thread+0xc5

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J 6283 sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V (0 bytes) @ 0x00007f4960aabb67 [0x00007f4960aabb00+0x67] j net.openhft.chronicle.core.UnsafeMemory.copyMemory0(Ljava/lang/Object;JLjava/lang/Object;JJ)V+28 J 25380 C2 net.openhft.chronicle.queue.ExcerptAppender.writeBytes(Lnet/openhft/chronicle/bytes/Bytes;)V (8 bytes) @ 0x00007f49642c79c4 [0x00007f49642c6e80+0xb44] J 23986 C2 com.webaction.runtime.BufferManagement.BufferManagerWithChronicleQueue.put(Lcom/webaction/proc/events/WAEvent;)V (312 bytes) @ 0x00007f496373326c [0x00007f4963731ba0+0x16cc] J 23891 C2 com.webaction.proc.BufferBasedTxnDetails.addEvent(Ljava/lang/Comparable;Lcom/webaction/proc/events/WAEvent;)Ljava/lang/Long; (53 bytes) @ 0x00007f4963de7610 [0x00007f4963de75a0+0x70] J 25709 C2 com.webaction.proc.TxnCache.update(Lcom/webaction/proc/events/WAEvent;)V (1595 bytes) @ 0x00007f496453a138 [0x00007f4964537b20+0x2618] J 22850 C2 com.webaction.proc.OracleReader_1_0.receiveImpl(ILcom/webaction/event/Event;)V (762 bytes) @ 0x00007f4962e6574c [0x00007f4962e655e0+0x16c] J 23700% C2 com.webaction.runtime.components.Source.run()V (203 bytes) @ 0x00007f4963ca7298 [0x00007f4963ca7120+0x178] J 6877 C1 java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (14 bytes) @ 0x00007f49614a988c [0x00007f49614a9780+0x10c] J 17246 C1 java.util.concurrent.FutureTask.run()V (126 bytes) @ 0x00007f4962d5e0c4 [0x00007f4962d5de80+0x244] j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Any ideas ?

[dpisklov]

What it has to do with Chronicle map? In the stack trace you have Chronicle Queue failing to write some data, and my guess is it's because it runs out of disk space.

J 6283 sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V (0 bytes) @ 0x00007f4960aabbe1 [0x00007f4960aabb00+0xe1] j net.openhft.chronicle.core.UnsafeMemory.copyMemory0( Ljava/lang/Object;JLjava/lang/Object;JJ)V+28 J 25380 C2 net.openhft.chronicle.queue.ExcerptAppender.writeBytes( Lnet/openhft/chronicle/bytes/Bytes;)V (8 bytes) @ 0x00007f49642c79c4 [0x00007f49642c6e80+0xb44]

[bhushankh]

Thanks for taking a look. The partition we are using has 16TB free. Will double check and see if that is an issue.

[dpisklov]

What file system is that partition?

[anirudh-striim]

The storage is xfs.

[anirudh-striim]

And it is not running out of disk space. We changed the storage to a larger disk and hit the same issue.

[dpisklov]

Superseded by the issue in Chronicle-Queue