search

kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 328 forks source link

Multiple kamon instances on linux - crashing the JVM #444

Closed kcris closed 7 years ago

kcris commented 7 years ago

Hi, I have a scala (2.11.7) app that runs on linux and uses akka (2.4.14) + kamon (0.6.3). One instance runs fine, kamon is happy and all.

As soon as I start the second instance with kamon enabled, the jvm crashes. The java vm itself is aborted with a core dump (not my application). Any number of instances other than 1, results in the same behavior.

The same app without kamon, runs perfectly with many instances on linux. Also, tried multiple instances on Windows, I get no crash with kamon enabled.

The generated log shows indeed this has something to do with kamon.

See below the os/environment details

scala 2.11.7
akka 2.4.14
kamon 0.6.3
uname-a
***KERNEL***
Linux empty-srv 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/issue
***OS***
Ubuntu 14.04.1 LTS \n \l
java -version
***JAVA***
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

crash log

hs_err_pid1929.log

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000009d6e, pid=1929, tid=0x00007f4e381ef700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x0000000000009d6e
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007f4e24065800):  JavaThread "kamon-kamon.system-metrics.sigar-dispatcher-6" [_thread_in_native, id=1967, stack(0x00007f4e380ef000,0x00007f4e381f0000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000009d6e

Registers:
RAX=0x00007f4e18011050, RBX=0x00007f4e381edbb0, RCX=0x00007f4e18011050, RDX=0x0000000000000000
RSP=0x00007f4e381edab8, RBP=0x00007f4e240659f8, RSI=0x00007f4e381edac0, RDI=0x00007f4e18004560
R8 =0x0000000000000006, R9 =0x00000000c3716a70, R10=0x0000000000000162, R11=0x00007f4e8afc39e8
R12=0x00007f4e18011050, R13=0x00007f4e381edbb8, R14=0x00007f4e18000a30, R15=0x00007f4e24065800
RIP=0x0000000000009d6e, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f4e381edab8)
0x00007f4e381edab8:   00007f4e381fc7d8 00007f4e382199d3
0x00007f4e381edac8:   00000001003796d0 00007f4e24065800
0x00007f4e381edad8:   0000000000000000 00000001003796d0
0x00007f4e381edae8:   00007f4e24065800 00007f4e381edba0
0x00007f4e381edaf8:   00007f4e8a6e7d0f 00007f4e381edb20
0x00007f4e381edb08:   00007f4e381edb40 00007f4e31f68c60
0x00007f4e381edb18:   00007f4e381edb90 0000000000000000
0x00007f4e381edb28:   00007f4e31f68c58 00007f4e381edbb8
0x00007f4e381edb38:   00007f4e75017a34 00007f4e381edba0
0x00007f4e381edb48:   00007f4e750082bd 00007f4e381edb50
0x00007f4e381edb58:   0000000000000000 00007f4e381edbb8
0x00007f4e381edb68:   00007f4e31f69718 0000000000000000
0x00007f4e381edb78:   00007f4e31f68c60 0000000000000000
0x00007f4e381edb88:   00007f4e381edbb0 00007f4e381edc00
0x00007f4e381edb98:   00007f4e750082bd 0000000000000000
0x00007f4e381edba8:   00007f4e75011538 00000000c3716a70
0x00007f4e381edbb8:   00000000f1efd9a0 00007f4e381edbc0
0x00007f4e381edbc8:   00007f4e31f68d02 00007f4e381edc18
0x00007f4e381edbd8:   00007f4e31f69718 0000000000000000
0x00007f4e381edbe8:   00007f4e31f68d30 00007f4e381edbb0
0x00007f4e381edbf8:   00007f4e381edc18 00007f4e381edc60
0x00007f4e381edc08:   00007f4e75008040 00000000f1efd9a0
0x00007f4e381edc18:   00000000c3716a70 00007f4e381edc20
0x00007f4e381edc28:   00007f4e3861c5d9 00007f4e381edc70
0x00007f4e381edc38:   00007f4e332cb380 0000000000000000
0x00007f4e381edc48:   00007f4e3861c5f8 00007f4e381edc18
0x00007f4e381edc58:   00007f4e381edc70 00007f4e381edcc0
0x00007f4e381edc68:   00007f4e75008040 00000000c3716a70
0x00007f4e381edc78:   00000000c3716a70 00007f4e381edc80
0x00007f4e381edc88:   00007f4e3861c7f1 00007f4e381edcd8
0x00007f4e381edc98:   00007f4e332cb380 0000000000000000
0x00007f4e381edca8:   00007f4e3861c838 00007f4e381edc70 

Instructions: (pc=0x0000000000009d6e)
0x0000000000009d4e:   
[error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xb]

Register to memory mapping:

RAX=0x00007f4e18011050 is an unknown value
RBX=0x00007f4e381edbb0 is pointing into the stack for thread: 0x00007f4e24065800
RCX=0x00007f4e18011050 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x00007f4e381edab8 is pointing into the stack for thread: 0x00007f4e24065800
RBP=0x00007f4e240659f8 is an unknown value
RSI=0x00007f4e381edac0 is pointing into the stack for thread: 0x00007f4e24065800
RDI=0x00007f4e18004560 is an unknown value
R8 =0x0000000000000006 is an unknown value
R9 =0x00000000c3716a70 is an oop
org.hyperic.sigar.Sigar 
 - klass: 'org/hyperic/sigar/Sigar'
R10=0x0000000000000162 is an unknown value
R11=0x00007f4e8afc39e8: <offset 0xfc59e8> in /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so at 0x00007f4e89ffe000
R12=0x00007f4e18011050 is an unknown value
R13=0x00007f4e381edbb8 is pointing into the stack for thread: 0x00007f4e24065800
R14=0x00007f4e18000a30 is an unknown value
R15=0x00007f4e24065800 is a thread

Stack: [0x00007f4e380ef000,0x00007f4e381f0000],  sp=0x00007f4e381edab8,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0000000000009d6e

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.hyperic.sigar.Cpu.gather(Lorg/hyperic/sigar/Sigar;)V+0
j  org.hyperic.sigar.Cpu.fetch(Lorg/hyperic/sigar/Sigar;)Lorg/hyperic/sigar/Cpu;+10
j  org.hyperic.sigar.Sigar.getCpu()Lorg/hyperic/sigar/Cpu;+1
j  org.hyperic.sigar.Sigar.getCpuPerc()Lorg/hyperic/sigar/CpuPerc;+25
j  kamon.system.sigar.CpuMetrics.kamon$system$sigar$CpuMetrics$$cpuPerc$1()Lscala/Tuple5;+4
j  kamon.system.sigar.CpuMetrics$$anonfun$1.apply()Lscala/Tuple5;+4
j  kamon.system.sigar.CpuMetrics$$anonfun$1.apply()Ljava/lang/Object;+1
J 4529 C1 kamon.system.sigar.SigarSafeRunner$.runSafe(Lscala/Function0;Lscala/Function0;Ljava/lang/String;Lakka/event/LoggingAdapter;)Ljava/lang/Object; (57 bytes) @ 0x00007f4e752922bc [0x00007f4e752921a0+0x11c]
j  kamon.system.sigar.CpuMetrics.update()V+25
j  kamon.system.sigar.SigarMetricsUpdater.updateMetrics()V+23
j  kamon.system.sigar.SigarMetricsUpdater$$anonfun$receive$1.applyOrElse(Ljava/lang/Object;Lscala/Function1;)Ljava/lang/Object;+14
j  akka.actor.Actor$class.aroundReceive(Lakka/actor/Actor;Lscala/PartialFunction;Ljava/lang/Object;)V+10
j  kamon.system.sigar.SigarMetricsUpdater.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V+3
j  akka.actor.ActorCell.receiveMessage(Ljava/lang/Object;)V+15
j  akka.actor.ActorCell.invoke_aroundBody0(Lakka/actor/ActorCell;Lakka/dispatch/Envelope;Lorg/aspectj/lang/JoinPoint;)V+59
j  akka.actor.ActorCell$AjcClosure1.run([Ljava/lang/Object;)Ljava/lang/Object;+23
J 4163 C1 org.aspectj.runtime.reflect.JoinPointImpl.proceed()Ljava/lang/Object; (24 bytes) @ 0x00007f4e751249ac [0x00007f4e751247c0+0x1ec]
j  akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply()Ljava/lang/Object;+4
J 4356 C1 kamon.trace.Tracer$.withContext(Lkamon/trace/TraceContext;Lscala/Function0;)Ljava/lang/Object; (47 bytes) @ 0x00007f4e7535d3fc [0x00007f4e7535cd00+0x6fc]
j  akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(Lorg/aspectj/lang/ProceedingJoinPoint;Lakka/kamon/instrumentation/EnvelopeContext;)Ljava/lang/Object;+16
j  akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(Lorg/aspectj/lang/ProceedingJoinPoint;Lakka/actor/ActorCell;Lakka/dispatch/Envelope;)Ljava/lang/Object;+15
j  akka.actor.ActorCell.invoke(Lakka/dispatch/Envelope;)V+59
J 4652 C1 akka.dispatch.Mailbox.processMailbox(IJ)V (100 bytes) @ 0x00007f4e755e1e24 [0x00007f4e755e1780+0x6a4]
J 4489 C1 akka.dispatch.Mailbox.run()V (59 bytes) @ 0x00007f4e755ac88c [0x00007f4e755abb00+0xd8c]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f4e1400d800 JavaThread "ForkJoinPool-3-worker-7" daemon [_thread_blocked, id=2007, stack(0x00007f4e31112000,0x00007f4e31213000)]
  0x00007f4e10019000 JavaThread "pool-2-thread-3" [_thread_blocked, id=2006, stack(0x00007f4e31213000,0x00007f4e31314000)]
  0x00007f4e00066000 JavaThread "pool-2-thread-2" [_thread_blocked, id=2005, stack(0x00007f4e31314000,0x00007f4e31415000)]
  0x00007f4e0005b800 JavaThread "AMQP Connection 192.168.56.132:5674" [_thread_in_native, id=2004, stack(0x00007f4e31415000,0x00007f4e31516000)]
  0x00007f4e0c04d800 JavaThread "ForkJoinPool-3-worker-1" daemon [_thread_blocked, id=2003, stack(0x00007f4e31516000,0x00007f4e31617000)]
  0x00007f4e0c04c000 JavaThread "ForkJoinPool-3-worker-3" daemon [_thread_blocked, id=2002, stack(0x00007f4e32b79000,0x00007f4e32c7a000)]
  0x00007f4e0000d800 JavaThread "JobsConsolidator-akka.io.pinned-dispatcher-14" [_thread_in_native, id=2001, stack(0x00007f4e31817000,0x00007f4e31918000)]
  0x00007f4e10001000 JavaThread "ForkJoinPool-3-worker-5" daemon [_thread_blocked, id=2000, stack(0x00007f4e31d18000,0x00007f4e31e19000)]
  0x00007f4e04086000 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-13" [_thread_blocked, id=1985, stack(0x00007f4e32876000,0x00007f4e32977000)]
  0x00007f4e0408a000 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-11" [_thread_blocked, id=1983, stack(0x00007f4e32e99000,0x00007f4e32f9a000)]
  0x00007f4e08001000 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-10" [_thread_blocked, id=1982, stack(0x00007f4e32f9a000,0x00007f4e3309b000)]
  0x00007f4e14006000 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-9" [_thread_blocked, id=1981, stack(0x00007f4e3309b000,0x00007f4e3319c000)]
  0x00007f4e20049000 JavaThread "kamon-akka.actor.default-dispatcher-11" [_thread_blocked, id=1980, stack(0x00007f4e334f5000,0x00007f4e335f6000)]
  0x00007f4e24097800 JavaThread "kamon-akka.actor.default-dispatcher-10" [_thread_blocked, id=1979, stack(0x00007f4e335f6000,0x00007f4e336f7000)]
  0x00007f4e2c018800 JavaThread "kamon-akka.actor.default-dispatcher-9" [_thread_blocked, id=1978, stack(0x00007f4e336f7000,0x00007f4e337f8000)]
  0x00007f4e28003000 JavaThread "kamon-akka.actor.default-dispatcher-8" [_thread_blocked, id=1977, stack(0x00007f4e337f8000,0x00007f4e338f9000)]
  0x00007f4e8400c000 JavaThread "DestroyJavaVM" [_thread_blocked, id=1934, stack(0x00007f4e8bb04000,0x00007f4e8bc05000)]
  0x00007f4e14003000 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-8" [_thread_blocked, id=1976, stack(0x00007f4e32977000,0x00007f4e32a78000)]
  0x00007f4e8501a800 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-3" [_thread_blocked, id=1971, stack(0x00007f4e339fa000,0x00007f4e33afb000)]
  0x00007f4e85018800 JavaThread "JobsConsolidator-akka.actor.default-dispatcher-2" [_thread_blocked, id=1970, stack(0x00007f4e33afb000,0x00007f4e33bfc000)]
  0x00007f4e85017800 JavaThread "JobsConsolidator-scheduler-1" [_thread_blocked, id=1969, stack(0x00007f4e33bfc000,0x00007f4e33cfd000)]
  0x00007f4e24096000 JavaThread "kamon-kamon.system-metrics.context-switches-dispatcher-7" [_thread_blocked, id=1968, stack(0x00007f4e3319c000,0x00007f4e3329d000)]
=>0x00007f4e24065800 JavaThread "kamon-kamon.system-metrics.sigar-dispatcher-6" [_thread_in_native, id=1967, stack(0x00007f4e380ef000,0x00007f4e381f0000)]
  0x00007f4e28001000 JavaThread "kamon-akka.actor.default-dispatcher-5" [_thread_blocked, id=1966, stack(0x00007f4e33cfd000,0x00007f4e33dfe000)]
  0x00007f4e84d79800 JavaThread "kamon-akka.actor.default-dispatcher-4" [_thread_blocked, id=1965, stack(0x00007f4e33dfe000,0x00007f4e33eff000)]
  0x00007f4e84d8b800 JavaThread "kamon-akka.actor.default-dispatcher-3" [_thread_blocked, id=1964, stack(0x00007f4e33eff000,0x00007f4e34000000)]
  0x00007f4e84d6f800 JavaThread "kamon-akka.actor.default-dispatcher-2" [_thread_blocked, id=1963, stack(0x00007f4e38327000,0x00007f4e38428000)]
  0x00007f4e84c4f800 JavaThread "kamon-scheduler-1" [_thread_in_vm, id=1955, stack(0x00007f4e70058000,0x00007f4e70159000)]
  0x00007f4e84396000 JavaThread "Service Thread" daemon [_thread_blocked, id=1946, stack(0x00007f4e70b06000,0x00007f4e70c07000)]
  0x00007f4e8438d800 JavaThread "C1 CompilerThread2" daemon [_thread_blocked, id=1945, stack(0x00007f4e70c07000,0x00007f4e70d08000)]
  0x00007f4e8437b800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=1944, stack(0x00007f4e70d08000,0x00007f4e70e09000)]
  0x00007f4e8437b000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=1943, stack(0x00007f4e70e09000,0x00007f4e70f0a000)]
  0x00007f4e8417f800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1942, stack(0x00007f4e71134000,0x00007f4e71235000)]
  0x00007f4e8414d800 JavaThread "Finalizer" daemon [_thread_blocked, id=1941, stack(0x00007f4e71235000,0x00007f4e71336000)]
  0x00007f4e84149000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1940, stack(0x00007f4e71336000,0x00007f4e71437000)]

Other Threads:
  0x00007f4e84141000 VMThread [stack: 0x00007f4e71437000,0x00007f4e71538000] [id=1939]
  0x00007f4e84398800 WatcherThread [stack: 0x00007f4e70a05000,0x00007f4e70b06000] [id=1947]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
 PSYoungGen      total 289792K, used 125069K [0x00000000eb600000, 0x00000000ff080000, 0x0000000100000000)
  eden space 258560K, 41% used [0x00000000eb600000,0x00000000f1f9f1b8,0x00000000fb280000)
  from space 31232K, 54% used [0x00000000fd200000,0x00000000fe284598,0x00000000ff080000)
  to   space 31744K, 0% used [0x00000000fb280000,0x00000000fb280000,0x00000000fd180000)
 ParOldGen       total 84480K, used 53594K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 63% used [0x00000000c2200000,0x00000000c56568f8,0x00000000c7480000)
 Metaspace       used 43556K, capacity 43850K, committed 44160K, reserved 1087488K
  class space    used 5655K, capacity 5745K, committed 5760K, reserved 1048576K

Card table byte_map: [0x00007f4e884a5000,0x00007f4e88695000] byte_map_base: 0x00007f4e87e94000

Marking Bits: (ParMarkBitMap*) 0x00007f4e8afc2c80
 Begin Bits: [0x00007f4e7300f000, 0x00007f4e73f87000)
 End Bits:   [0x00007f4e73f87000, 0x00007f4e74eff000)

Polling page: 0x00007f4e8bc0d000

CodeCache: size=245760Kb used=14548Kb max_used=15851Kb free=231211Kb
 bounds [0x00007f4e75000000, 0x00007f4e76060000, 0x00007f4e84000000]
 total_blobs=4606 nmethods=4072 adapters=446
 compilation: enabled

Compilation events (10 events):
Event: 98.592 Thread 0x00007f4e8438d800 4691       3       scala.Tuple2$mcJJ$sp::_1$mcJ$sp (5 bytes)
Event: 98.592 Thread 0x00007f4e8438d800 nmethod 4691 0x00007f4e75cdbe90 code [0x00007f4e75cdbfe0, 0x00007f4e75cdc130]
Event: 98.592 Thread 0x00007f4e8438d800 4692       3       scala.Tuple2$mcJJ$sp::_2$mcJ$sp (5 bytes)
Event: 98.592 Thread 0x00007f4e8438d800 nmethod 4692 0x00007f4e75cdbb50 code [0x00007f4e75cdbca0, 0x00007f4e75cdbdf0]
Event: 99.571 Thread 0x00007f4e8438d800 4693       3       scala.collection.convert.Decorators$AsScala::<init> (21 bytes)
Event: 99.572 Thread 0x00007f4e8438d800 nmethod 4693 0x00007f4e752b7c90 code [0x00007f4e752b7e00, 0x00007f4e752b8030]
Event: 99.572 Thread 0x00007f4e8438d800 4694       3       scala.collection.convert.Decorators$AsScala::asScala (10 bytes)
Event: 99.572 Thread 0x00007f4e8438d800 nmethod 4694 0x00007f4e7555fc50 code [0x00007f4e7555fdc0, 0x00007f4e7555ffe8]
Event: 99.572 Thread 0x00007f4e8437b000 4695       4       scala.collection.convert.Wrappers$JIteratorWrapper::hasNext (10 bytes)
Event: 99.572 Thread 0x00007f4e8437b000 nmethod 4695 0x00007f4e7524b710 code [0x00007f4e7524b860, 0x00007f4e7524b918]

GC Heap History (10 events):
Event: 2.791 GC heap before
{Heap before GC invocations=10 (full 1):
 PSYoungGen      total 169472K, used 169459K [0x00000000eb600000, 0x00000000f9e80000, 0x0000000100000000)
  eden space 151552K, 100% used [0x00000000eb600000,0x00000000f4a00000,0x00000000f4a00000)
  from space 17920K, 99% used [0x00000000f6080000,0x00000000f71fcfd0,0x00000000f7200000)
  to   space 23040K, 0% used [0x00000000f4a00000,0x00000000f4a00000,0x00000000f6080000)
 ParOldGen       total 43520K, used 26004K [0x00000000c2200000, 0x00000000c4c80000, 0x00000000eb600000)
  object space 43520K, 59% used [0x00000000c2200000,0x00000000c3b651c8,0x00000000c4c80000)
 Metaspace       used 30557K, capacity 30682K, committed 30848K, reserved 1077248K
  class space    used 3907K, capacity 3964K, committed 3968K, reserved 1048576K
Event: 2.812 GC heap after
Heap after GC invocations=10 (full 1):
 PSYoungGen      total 174592K, used 23008K [0x00000000eb600000, 0x00000000fbb80000, 0x0000000100000000)
  eden space 151552K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000f4a00000)
  from space 23040K, 99% used [0x00000000f4a00000,0x00000000f6078398,0x00000000f6080000)
  to   space 30720K, 0% used [0x00000000f9d80000,0x00000000f9d80000,0x00000000fbb80000)
 ParOldGen       total 43520K, used 35257K [0x00000000c2200000, 0x00000000c4c80000, 0x00000000eb600000)
  object space 43520K, 81% used [0x00000000c2200000,0x00000000c446e7a0,0x00000000c4c80000)
 Metaspace       used 30557K, capacity 30682K, committed 30848K, reserved 1077248K
  class space    used 3907K, capacity 3964K, committed 3968K, reserved 1048576K
}
Event: 2.812 GC heap before
{Heap before GC invocations=11 (full 2):
 PSYoungGen      total 174592K, used 23008K [0x00000000eb600000, 0x00000000fbb80000, 0x0000000100000000)
  eden space 151552K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000f4a00000)
  from space 23040K, 99% used [0x00000000f4a00000,0x00000000f6078398,0x00000000f6080000)
  to   space 30720K, 0% used [0x00000000f9d80000,0x00000000f9d80000,0x00000000fbb80000)
 ParOldGen       total 43520K, used 35257K [0x00000000c2200000, 0x00000000c4c80000, 0x00000000eb600000)
  object space 43520K, 81% used [0x00000000c2200000,0x00000000c446e7a0,0x00000000c4c80000)
 Metaspace       used 30557K, capacity 30682K, committed 30848K, reserved 1077248K
  class space    used 3907K, capacity 3964K, committed 3968K, reserved 1048576K
Event: 2.887 GC heap after
Heap after GC invocations=11 (full 2):
 PSYoungGen      total 174592K, used 0K [0x00000000eb600000, 0x00000000fbb80000, 0x0000000100000000)
  eden space 151552K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000f4a00000)
  from space 23040K, 0% used [0x00000000f4a00000,0x00000000f4a00000,0x00000000f6080000)
  to   space 30720K, 0% used [0x00000000f9d80000,0x00000000f9d80000,0x00000000fbb80000)
 ParOldGen       total 84480K, used 40920K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49f6100,0x00000000c7480000)
 Metaspace       used 30557K, capacity 30682K, committed 30848K, reserved 1077248K
  class space    used 3907K, capacity 3964K, committed 3968K, reserved 1048576K
}
Event: 3.482 GC heap before
{Heap before GC invocations=12 (full 2):
 PSYoungGen      total 174592K, used 151552K [0x00000000eb600000, 0x00000000fbb80000, 0x0000000100000000)
  eden space 151552K, 100% used [0x00000000eb600000,0x00000000f4a00000,0x00000000f4a00000)
  from space 23040K, 0% used [0x00000000f4a00000,0x00000000f4a00000,0x00000000f6080000)
  to   space 30720K, 0% used [0x00000000f9d80000,0x00000000f9d80000,0x00000000fbb80000)
 ParOldGen       total 84480K, used 40920K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49f6100,0x00000000c7480000)
 Metaspace       used 36051K, capacity 36382K, committed 36864K, reserved 1081344K
  class space    used 4639K, capacity 4751K, committed 4864K, reserved 1048576K
Event: 3.497 GC heap after
Heap after GC invocations=12 (full 2):
 PSYoungGen      total 219648K, used 18441K [0x00000000eb600000, 0x00000000fb380000, 0x0000000100000000)
  eden space 197120K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000f7680000)
  from space 22528K, 81% used [0x00000000f9d80000,0x00000000faf82550,0x00000000fb380000)
  to   space 31232K, 0% used [0x00000000f7680000,0x00000000f7680000,0x00000000f9500000)
 ParOldGen       total 84480K, used 40928K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49f8100,0x00000000c7480000)
 Metaspace       used 36051K, capacity 36382K, committed 36864K, reserved 1081344K
  class space    used 4639K, capacity 4751K, committed 4864K, reserved 1048576K
}
Event: 3.986 GC heap before
{Heap before GC invocations=13 (full 2):
 PSYoungGen      total 219648K, used 215561K [0x00000000eb600000, 0x00000000fb380000, 0x0000000100000000)
  eden space 197120K, 100% used [0x00000000eb600000,0x00000000f7680000,0x00000000f7680000)
  from space 22528K, 81% used [0x00000000f9d80000,0x00000000faf82550,0x00000000fb380000)
  to   space 31232K, 0% used [0x00000000f7680000,0x00000000f7680000,0x00000000f9500000)
 ParOldGen       total 84480K, used 40928K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49f8100,0x00000000c7480000)
 Metaspace       used 38463K, capacity 38750K, committed 39168K, reserved 1083392K
  class space    used 4924K, capacity 5007K, committed 5120K, reserved 1048576K
Event: 4.006 GC heap after
Heap after GC invocations=13 (full 2):
 PSYoungGen      total 228352K, used 21274K [0x00000000eb600000, 0x00000000ff200000, 0x0000000100000000)
  eden space 197120K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000f7680000)
  from space 31232K, 68% used [0x00000000f7680000,0x00000000f8b468b8,0x00000000f9500000)
  to   space 32768K, 0% used [0x00000000fd200000,0x00000000fd200000,0x00000000ff200000)
 ParOldGen       total 84480K, used 40936K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49fa100,0x00000000c7480000)
 Metaspace       used 38463K, capacity 38750K, committed 39168K, reserved 1083392K
  class space    used 4924K, capacity 5007K, committed 5120K, reserved 1048576K
}
Event: 4.590 GC heap before
{Heap before GC invocations=14 (full 2):
 PSYoungGen      total 228352K, used 218394K [0x00000000eb600000, 0x00000000ff200000, 0x0000000100000000)
  eden space 197120K, 100% used [0x00000000eb600000,0x00000000f7680000,0x00000000f7680000)
  from space 31232K, 68% used [0x00000000f7680000,0x00000000f8b468b8,0x00000000f9500000)
  to   space 32768K, 0% used [0x00000000fd200000,0x00000000fd200000,0x00000000ff200000)
 ParOldGen       total 84480K, used 40936K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 48% used [0x00000000c2200000,0x00000000c49fa100,0x00000000c7480000)
 Metaspace       used 40910K, capacity 41182K, committed 41600K, reserved 1085440K
  class space    used 5282K, capacity 5391K, committed 5504K, reserved 1048576K
Event: 4.611 GC heap after
Heap after GC invocations=14 (full 2):
 PSYoungGen      total 289792K, used 16913K [0x00000000eb600000, 0x00000000ff080000, 0x0000000100000000)
  eden space 258560K, 0% used [0x00000000eb600000,0x00000000eb600000,0x00000000fb280000)
  from space 31232K, 54% used [0x00000000fd200000,0x00000000fe284598,0x00000000ff080000)
  to   space 31744K, 0% used [0x00000000fb280000,0x00000000fb280000,0x00000000fd180000)
 ParOldGen       total 84480K, used 53594K [0x00000000c2200000, 0x00000000c7480000, 0x00000000eb600000)
  object space 84480K, 63% used [0x00000000c2200000,0x00000000c56568f8,0x00000000c7480000)
 Metaspace       used 40910K, capacity 41182K, committed 41600K, reserved 1085440K
  class space    used 5282K, capacity 5391K, committed 5504K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 8.931 Thread 0x00007f4e10019000 Uncommon trap: reason=bimorphic action=maybe_recompile pc=0x00007f4e75411e98 method=java.util.AbstractCollection.isEmpty()Z @ 1
Event: 9.380 Thread 0x00007f4e84d79800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e75157ddc method=scala.concurrent.forkjoin.ForkJoinPool.scan(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)Lscala/concurrent/forkjoin/ForkJoinTask; @ 410
Event: 13.682 Thread 0x00007f4e84d79800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e7554c13c method=java.lang.Math.max(JJ)J @ 3
Event: 42.532 Thread 0x00007f4e20049000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e75145118 method=scala.concurrent.forkjoin.ForkJoinPool.scan(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)Lscala/concurrent/forkjoin/ForkJoinTask; @ 182
Event: 42.532 Thread 0x00007f4e84c4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e7520b6fc method=scala.concurrent.forkjoin.ForkJoinPool.signalWork(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)V @ 141
Event: 44.761 Thread 0x00007f4e84d8b800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e75d09218 method=scala.concurrent.forkjoin.ForkJoinPool.scan(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)Lscala/concurrent/forkjoin/ForkJoinTask; @ 312
Event: 51.633 Thread 0x00007f4e84c4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e757a4208 method=scala.concurrent.forkjoin.ForkJoinPool.signalWork(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)V @ 141
Event: 51.633 Thread 0x00007f4e28003000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e752ccbfc method=scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;JJ)V @ 89
Event: 51.635 Thread 0x00007f4e24097800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e752ccad4 method=scala.concurrent.forkjoin.ForkJoinPool.scan(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)Lscala/concurrent/forkjoin/ForkJoinTask; @ 638
Event: 65.271 Thread 0x00007f4e24097800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f4e75769540 method=scala.concurrent.duration.Duration$.fromNanos(J)Lscala/concurrent/duration/FiniteDuration; @ 107

Internal exceptions (10 events):
Event: 2.686 Thread 0x00007f4e0c003000 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x00000000f2f431a0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605]
Event: 2.686 Thread 0x00007f4e0c003000 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x00000000f2f437a8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605]
Event: 2.686 Thread 0x00007f4e0c003000 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x00000000f2f43e18) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605]
Event: 2.686 Thread 0x00007f4e0c003000 Exception <a 'java/lang/ArrayIndexOutOfBoundsException'> (0x00000000f2f44420) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605]
Event: 3.290 Thread 0x00007f4e8501c000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000f15c9ca8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/methodHandles.cpp, line 1146]
Event: 3.291 Thread 0x00007f4e8501c000 Exception <a 'java/lang/NoSuchFieldError': method resolution failed> (0x00000000f15d7fd8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/methodHandles.cpp, line 1146]
Event: 28.818 Thread 0x00007f4e0005b800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000000f045e4f8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 43.850 Thread 0x00007f4e0005b800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000000f045e9d0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 58.865 Thread 0x00007f4e0005b800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000000f045ec28) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 88.816 Thread 0x00007f4e0005b800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000000f045fa58) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]

Events (10 events):
Event: 73.798 Executing VM operation: RevokeBias done
Event: 73.798 Executing VM operation: RevokeBias
Event: 73.798 Executing VM operation: RevokeBias done
Event: 73.799 Executing VM operation: RevokeBias
Event: 73.799 Executing VM operation: RevokeBias done
Event: 82.399 Thread 0x00007f4e8438d800 flushing nmethod 0x00007f4e75834fd0
Event: 82.655 Thread 0x00007f4e8437b000 flushing nmethod 0x00007f4e75ce1010
Event: 83.592 Thread 0x00007f4e8438d800 flushing nmethod 0x00007f4e75dfbdd0
Event: 97.662 Thread 0x00007f4e84d6f800 DEOPT PACKING pc=0x00007f4e75216adb sp=0x00007f4e384265a0
Event: 97.662 Thread 0x00007f4e84d6f800 DEOPT UNPACKING pc=0x00007f4e75047633 sp=0x00007f4e38426318 mode 0

Dynamic libraries:
00400000-00401000 r-xp 00000000 fc:00 396300                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00600000-00601000 rw-p 00000000 fc:00 396300                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00f9e000-00fbf000 rw-p 00000000 00:00 0                                  [heap]
c2200000-c7480000 rw-p 00000000 00:00 0 
c7480000-eb600000 ---p 00000000 00:00 0 
eb600000-ff080000 rw-p 00000000 00:00 0 
ff080000-100000000 ---p 00000000 00:00 0 
100000000-1005a0000 rw-p 00000000 00:00 0 
1005a0000-140000000 ---p 00000000 00:00 0 
7f4df0000000-7f4df0021000 rw-p 00000000 00:00 0 
7f4df0021000-7f4df4000000 ---p 00000000 00:00 0 
7f4df4000000-7f4df4021000 rw-p 00000000 00:00 0 
7f4df4021000-7f4df8000000 ---p 00000000 00:00 0 
7f4df8000000-7f4df8057000 rw-p 00000000 00:00 0 
7f4df8057000-7f4dfc000000 ---p 00000000 00:00 0 
7f4dfc000000-7f4dfc492000 rw-p 00000000 00:00 0 
7f4dfc492000-7f4e00000000 ---p 00000000 00:00 0 
7f4e00000000-7f4e0008c000 rw-p 00000000 00:00 0 
7f4e0008c000-7f4e04000000 ---p 00000000 00:00 0 
7f4e04000000-7f4e04106000 rw-p 00000000 00:00 0 
7f4e04106000-7f4e08000000 ---p 00000000 00:00 0 
7f4e08000000-7f4e08021000 rw-p 00000000 00:00 0 
7f4e08021000-7f4e0c000000 ---p 00000000 00:00 0 
7f4e0c000000-7f4e0c056000 rw-p 00000000 00:00 0 
7f4e0c056000-7f4e10000000 ---p 00000000 00:00 0 
7f4e10000000-7f4e10290000 rw-p 00000000 00:00 0 
7f4e10290000-7f4e14000000 ---p 00000000 00:00 0 
7f4e14000000-7f4e14021000 rw-p 00000000 00:00 0 
7f4e14021000-7f4e18000000 ---p 00000000 00:00 0 
7f4e18000000-7f4e1805b000 rw-p 00000000 00:00 0 
7f4e1805b000-7f4e1c000000 ---p 00000000 00:00 0 
7f4e1c000000-7f4e1c021000 rw-p 00000000 00:00 0 
7f4e1c021000-7f4e20000000 ---p 00000000 00:00 0 
7f4e20000000-7f4e20068000 rw-p 00000000 00:00 0 
7f4e20068000-7f4e24000000 ---p 00000000 00:00 0 
7f4e24000000-7f4e240c9000 rw-p 00000000 00:00 0 
7f4e240c9000-7f4e28000000 ---p 00000000 00:00 0 
7f4e28000000-7f4e28021000 rw-p 00000000 00:00 0 
7f4e28021000-7f4e2c000000 ---p 00000000 00:00 0 
7f4e2c000000-7f4e2c05e000 rw-p 00000000 00:00 0 
7f4e2c05e000-7f4e30000000 ---p 00000000 00:00 0 
7f4e31112000-7f4e31115000 ---p 00000000 00:00 0 
7f4e31115000-7f4e31213000 rw-p 00000000 00:00 0                          [stack:2007]
7f4e31213000-7f4e31216000 ---p 00000000 00:00 0 
7f4e31216000-7f4e31314000 rw-p 00000000 00:00 0                          [stack:2006]
7f4e31314000-7f4e31317000 ---p 00000000 00:00 0 
7f4e31317000-7f4e31415000 rw-p 00000000 00:00 0                          [stack:2005]
7f4e31415000-7f4e31418000 ---p 00000000 00:00 0 
7f4e31418000-7f4e31516000 rw-p 00000000 00:00 0                          [stack:2004]
7f4e31516000-7f4e31519000 ---p 00000000 00:00 0 
7f4e31519000-7f4e31797000 rw-p 00000000 00:00 0                          [stack:2003]
7f4e31797000-7f4e31817000 ---p 00000000 00:00 0 
7f4e31817000-7f4e3181a000 ---p 00000000 00:00 0 
7f4e3181a000-7f4e31b18000 rw-p 00000000 00:00 0                          [stack:2001]
7f4e31b18000-7f4e31d18000 rw-p 00000000 00:00 0 
7f4e31d18000-7f4e31d1b000 ---p 00000000 00:00 0 
7f4e31d1b000-7f4e32019000 rw-p 00000000 00:00 0                          [stack:2000]
7f4e32019000-7f4e3202f000 r-xp 00000000 fc:00 1573056                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4e3202f000-7f4e3222e000 ---p 00016000 fc:00 1573056                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4e3222e000-7f4e3222f000 rw-p 00015000 fc:00 1573056                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4e3222f000-7f4e32261000 r-xp 00000000 fc:00 530012                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libsunec.so
7f4e32261000-7f4e32460000 ---p 00032000 fc:00 530012                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libsunec.so
7f4e32460000-7f4e32466000 rw-p 00031000 fc:00 530012                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libsunec.so
7f4e32466000-7f4e32676000 rw-p 00000000 00:00 0 
7f4e32676000-7f4e32876000 rw-p 00000000 00:00 0 
7f4e32876000-7f4e32879000 ---p 00000000 00:00 0 
7f4e32879000-7f4e32977000 rw-p 00000000 00:00 0                          [stack:1985]
7f4e32977000-7f4e3297a000 ---p 00000000 00:00 0 
7f4e3297a000-7f4e32a78000 rw-p 00000000 00:00 0                          [stack:1976]
7f4e32a78000-7f4e32a7b000 ---p 00000000 00:00 0 
7f4e32a7b000-7f4e32b79000 rw-p 00000000 00:00 0 
7f4e32b79000-7f4e32b7c000 ---p 00000000 00:00 0 
7f4e32b7c000-7f4e32c7a000 rw-p 00000000 00:00 0                          [stack:2002]
7f4e32c7a000-7f4e32c7d000 ---p 00000000 00:00 0 
7f4e32c7d000-7f4e32d7b000 rw-p 00000000 00:00 0 
7f4e32d98000-7f4e32d9b000 ---p 00000000 00:00 0 
7f4e32d9b000-7f4e32e99000 rw-p 00000000 00:00 0 
7f4e32e99000-7f4e32e9c000 ---p 00000000 00:00 0 
7f4e32e9c000-7f4e32f9a000 rw-p 00000000 00:00 0                          [stack:1983]
7f4e32f9a000-7f4e32f9d000 ---p 00000000 00:00 0 
7f4e32f9d000-7f4e3309b000 rw-p 00000000 00:00 0                          [stack:1982]
7f4e3309b000-7f4e3309e000 ---p 00000000 00:00 0 
7f4e3309e000-7f4e3319c000 rw-p 00000000 00:00 0                          [stack:1981]
7f4e3319c000-7f4e3319f000 ---p 00000000 00:00 0 
7f4e3319f000-7f4e3349d000 rw-p 00000000 00:00 0                          [stack:1968]
7f4e334f5000-7f4e334f8000 ---p 00000000 00:00 0 
7f4e334f8000-7f4e335f6000 rw-p 00000000 00:00 0                          [stack:1980]
7f4e335f6000-7f4e335f9000 ---p 00000000 00:00 0 
7f4e335f9000-7f4e336f7000 rw-p 00000000 00:00 0                          [stack:1979]
7f4e336f7000-7f4e336fa000 ---p 00000000 00:00 0 
7f4e336fa000-7f4e337f8000 rw-p 00000000 00:00 0                          [stack:1978]
7f4e337f8000-7f4e337fb000 ---p 00000000 00:00 0 
7f4e337fb000-7f4e338f9000 rw-p 00000000 00:00 0                          [stack:1977]
7f4e338f9000-7f4e338fc000 ---p 00000000 00:00 0 
7f4e338fc000-7f4e339fa000 rw-p 00000000 00:00 0 
7f4e339fa000-7f4e339fd000 ---p 00000000 00:00 0 
7f4e339fd000-7f4e33afb000 rw-p 00000000 00:00 0                          [stack:1971]
7f4e33afb000-7f4e33afe000 ---p 00000000 00:00 0 
7f4e33afe000-7f4e33bfc000 rw-p 00000000 00:00 0                          [stack:1970]
7f4e33bfc000-7f4e33bff000 ---p 00000000 00:00 0 
7f4e33bff000-7f4e33cfd000 rw-p 00000000 00:00 0                          [stack:1969]
7f4e33cfd000-7f4e33d00000 ---p 00000000 00:00 0 
7f4e33d00000-7f4e33dfe000 rw-p 00000000 00:00 0                          [stack:1966]
7f4e33dfe000-7f4e33e01000 ---p 00000000 00:00 0 
7f4e33e01000-7f4e33eff000 rw-p 00000000 00:00 0                          [stack:1965]
7f4e33eff000-7f4e33f02000 ---p 00000000 00:00 0 
7f4e33f02000-7f4e34000000 rw-p 00000000 00:00 0                          [stack:1964]
7f4e34000000-7f4e34021000 rw-p 00000000 00:00 0 
7f4e34021000-7f4e38000000 ---p 00000000 00:00 0 
7f4e380de000-7f4e380e3000 r--s 002f9000 fc:00 396215                     /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar
7f4e380e3000-7f4e380e9000 r--s 0003f000 fc:00 396228                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar
7f4e380e9000-7f4e380eb000 r--s 00009000 fc:00 396227                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar
7f4e380eb000-7f4e380ef000 r--s 0003a000 fc:00 396226                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar
7f4e380ef000-7f4e380f2000 ---p 00000000 00:00 0 
7f4e380f2000-7f4e381f0000 rw-p 00000000 00:00 0                          [stack:1967]
7f4e381f0000-7f4e3821f000 r-xp 00000000 fc:00 786508                     /home/q/Consolidator/bin/native/libsigar-amd64-linux.so
7f4e3821f000-7f4e3831e000 ---p 0002f000 fc:00 786508                     /home/q/Consolidator/bin/native/libsigar-amd64-linux.so
7f4e3831e000-7f4e38324000 rw-p 0002e000 fc:00 786508                     /home/q/Consolidator/bin/native/libsigar-amd64-linux.so
7f4e38324000-7f4e38327000 rw-p 00000000 00:00 0 
7f4e38327000-7f4e3832a000 ---p 00000000 00:00 0 
7f4e3832a000-7f4e38628000 rw-p 00000000 00:00 0                          [stack:1963]
7f4e38628000-7f4e38828000 rw-p 00000000 00:00 0 
7f4e38828000-7f4e38a28000 rw-p 00000000 00:00 0 
7f4e38a28000-7f4e38c28000 rw-p 00000000 00:00 0 
7f4e38c28000-7f4e38e28000 rw-p 00000000 00:00 0 
7f4e38e28000-7f4e38e39000 r-xp 00000000 fc:00 529978                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnio.so
7f4e38e39000-7f4e39038000 ---p 00011000 fc:00 529978                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnio.so
7f4e39038000-7f4e39039000 rw-p 00010000 fc:00 529978                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnio.so
7f4e39039000-7f4e39239000 rw-p 00000000 00:00 0 
7f4e39239000-7f4e3c000000 rw-p 00000000 00:00 0 
7f4e3c000000-7f4e3c021000 rw-p 00000000 00:00 0 
7f4e3c021000-7f4e40000000 ---p 00000000 00:00 0 
7f4e40000000-7f4e40021000 rw-p 00000000 00:00 0 
7f4e40021000-7f4e44000000 ---p 00000000 00:00 0 
7f4e44000000-7f4e46989000 rw-p 00000000 00:00 0 
7f4e46989000-7f4e48000000 ---p 00000000 00:00 0 
7f4e48000000-7f4e49b3a000 rw-p 00000000 00:00 0 
7f4e49b3a000-7f4e4c000000 ---p 00000000 00:00 0 
7f4e4c000000-7f4e4c021000 rw-p 00000000 00:00 0 
7f4e4c021000-7f4e50000000 ---p 00000000 00:00 0 
7f4e50000000-7f4e53ffd000 rw-p 00000000 00:00 0 
7f4e53ffd000-7f4e54000000 ---p 00000000 00:00 0 
7f4e54000000-7f4e54021000 rw-p 00000000 00:00 0 
7f4e54021000-7f4e58000000 ---p 00000000 00:00 0 
7f4e58000000-7f4e58021000 rw-p 00000000 00:00 0 
7f4e58021000-7f4e5c000000 ---p 00000000 00:00 0 
7f4e5c000000-7f4e5c021000 rw-p 00000000 00:00 0 
7f4e5c021000-7f4e60000000 ---p 00000000 00:00 0 
7f4e60000000-7f4e60021000 rw-p 00000000 00:00 0 
7f4e60021000-7f4e64000000 ---p 00000000 00:00 0 
7f4e64000000-7f4e64021000 rw-p 00000000 00:00 0 
7f4e64021000-7f4e68000000 ---p 00000000 00:00 0 
7f4e68000000-7f4e68021000 rw-p 00000000 00:00 0 
7f4e68021000-7f4e6c000000 ---p 00000000 00:00 0 
7f4e6c000000-7f4e6c021000 rw-p 00000000 00:00 0 
7f4e6c021000-7f4e70000000 ---p 00000000 00:00 0 
7f4e70001000-7f4e70004000 r--s 0001a000 fc:00 396261                     /usr/lib/jvm/java-8-oracle/jre/lib/jce.jar
7f4e70004000-7f4e70008000 r--s 0009a000 fc:00 396270                     /usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar
7f4e70008000-7f4e70024000 r--s 00393000 fc:00 396222                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar
7f4e70024000-7f4e7002e000 r--s 00116000 fc:00 396221                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar
7f4e7002e000-7f4e70033000 r--s 00084000 fc:00 396288                     /usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar
7f4e70033000-7f4e70038000 r--s 002f9000 fc:00 396215                     /usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar
7f4e70038000-7f4e7003b000 r--s 0001a000 fc:00 396261                     /usr/lib/jvm/java-8-oracle/jre/lib/jce.jar
7f4e7003b000-7f4e7003f000 r--s 0009a000 fc:00 396270                     /usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar
7f4e7003f000-7f4e70058000 r--s 001af000 fc:00 786474                     /home/q/Consolidator/lib/aspectjweaver-1.8.9.jar
7f4e70058000-7f4e7005b000 ---p 00000000 00:00 0 
7f4e7005b000-7f4e70159000 rw-p 00000000 00:00 0                          [stack:1955]
7f4e70159000-7f4e70174000 r--s 001d3000 fc:00 396230                     /usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar
7f4e70174000-7f4e7018a000 r-xp 00000000 fc:00 529991                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnet.so
7f4e7018a000-7f4e7038a000 ---p 00016000 fc:00 529991                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnet.so
7f4e7038a000-7f4e7038b000 rw-p 00016000 fc:00 529991                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libnet.so
7f4e7038b000-7f4e7058b000 rw-p 00000000 00:00 0 
7f4e7058b000-7f4e7078b000 rw-p 00000000 00:00 0 
7f4e7078b000-7f4e70794000 r-xp 00000000 fc:00 530011                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libmanagement.so
7f4e70794000-7f4e70993000 ---p 00009000 fc:00 530011                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libmanagement.so
7f4e70993000-7f4e70994000 rw-p 00008000 fc:00 530011                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libmanagement.so
7f4e70994000-7f4e709a7000 r--s 00345000 fc:00 396265                     /usr/lib/jvm/java-8-oracle/jre/lib/resources.jar
7f4e709a7000-7f4e709b9000 r--s 000cf000 fc:00 786478                     /home/q/Consolidator/lib/consolidator-1.1-SNAPSHOT.jar
7f4e709b9000-7f4e709c0000 r--s 00048000 fc:00 786481                     /home/q/Consolidator/lib/httpcore-4.4.1.jar
7f4e709c0000-7f4e709c7000 r--s 0003a000 fc:00 786475                     /home/q/Consolidator/lib/commons-codec-1.9.jar
7f4e709c7000-7f4e709c9000 r--s 0000e000 fc:00 786476                     /home/q/Consolidator/lib/commons-logging-1.2.jar
7f4e709c9000-7f4e709d6000 r--s 000a4000 fc:00 786480                     /home/q/Consolidator/lib/httpclient-4.4.1.jar
7f4e709d6000-7f4e709dc000 r--s 00034000 fc:00 786489                     /home/q/Consolidator/lib/libthrift-0.9.3.jar
7f4e709dc000-7f4e709e6000 r--s 0005e000 fc:00 786473                     /home/q/Consolidator/lib/amqp-client-3.6.5.jar
7f4e709e6000-7f4e709e8000 r--s 0000c000 fc:00 786494                     /home/q/Consolidator/lib/mqbb-thrift-java-0.1.1-SNAPSHOT.jar
7f4e709e8000-7f4e709ec000 r--s 00041000 fc:00 786483                     /home/q/Consolidator/lib/jackson-core-2.8.4.jar
7f4e709ec000-7f4e709ef000 r--s 0000b000 fc:00 786482                     /home/q/Consolidator/lib/jackson-annotations-2.8.0.jar
7f4e709ef000-7f4e70a01000 r--s 0011c000 fc:00 786484                     /home/q/Consolidator/lib/jackson-databind-2.8.4.jar
7f4e70a01000-7f4e70a04000 r--s 00016000 fc:00 786500                     /home/q/Consolidator/lib/scala-logger-0.2-SNAPSHOT.jar
7f4e70a04000-7f4e70a05000 r--s 00009000 fc:00 786498                     /home/q/Consolidator/lib/scala-json-0.2-SNAPSHOT.jar
7f4e70a05000-7f4e70a06000 ---p 00000000 00:00 0 
7f4e70a06000-7f4e70b06000 rw-p 00000000 00:00 0                          [stack:1947]
7f4e70b06000-7f4e70b09000 ---p 00000000 00:00 0 
7f4e70b09000-7f4e70c07000 rw-p 00000000 00:00 0                          [stack:1946]
7f4e70c07000-7f4e70c0a000 ---p 00000000 00:00 0 
7f4e70c0a000-7f4e70d08000 rw-p 00000000 00:00 0                          [stack:1945]
7f4e70d08000-7f4e70d0b000 ---p 00000000 00:00 0 
7f4e70d0b000-7f4e70e09000 rw-p 00000000 00:00 0                          [stack:1944]
7f4e70e09000-7f4e70e0c000 ---p 00000000 00:00 0 
7f4e70e0c000-7f4e70f0a000 rw-p 00000000 00:00 0                          [stack:1943]
7f4e70f0a000-7f4e70f23000 r--s 001af000 fc:00 786474                     /home/q/Consolidator/lib/aspectjweaver-1.8.9.jar
7f4e70f23000-7f4e70f24000 r--s 00002000 fc:00 786487                     /home/q/Consolidator/lib/kamon-scala_2.11-0.6.3.jar
7f4e70f24000-7f4e70f28000 r--s 00018000 fc:00 786485                     /home/q/Consolidator/lib/kamon-akka_2.11-0.6.3.jar
7f4e70f28000-7f4e70f2f000 r--s 00266000 fc:00 786502                     /home/q/Consolidator/lib/sigar-loader-1.6.5-rev002.jar
7f4e70f2f000-7f4e70f33000 r--s 00020000 fc:00 786488                     /home/q/Consolidator/lib/kamon-system-metrics_2.11-0.6.3.jar
7f4e70f33000-7f4e70f36000 r--s 00019000 fc:00 786479                     /home/q/Consolidator/lib/HdrHistogram-2.1.8.jar
7f4e70f36000-7f4e70f42000 r--s 00093000 fc:00 786486                     /home/q/Consolidator/lib/kamon-core_2.11-0.6.3.jar
7f4e70f42000-7f4e70f47000 r--s 0003f000 fc:00 786477                     /home/q/Consolidator/lib/config-1.3.0.jar
7f4e70f47000-7f4e70f49000 r--s 0000d000 fc:00 786490                     /home/q/Consolidator/lib/log4j-1.2-api-2.7.jar
7f4e70f49000-7f4e70f64000 r--s 00122000 fc:00 786492                     /home/q/Consolidator/lib/log4j-core-2.7.jar
7f4e70f64000-7f4e70f69000 r--s 00031000 fc:00 786491                     /home/q/Consolidator/lib/log4j-api-2.7.jar
7f4e70f69000-7f4e70f6a000 r--s 00005000 fc:00 786493                     /home/q/Consolidator/lib/log4j-slf4j-impl-2.7.jar
7f4e70f6a000-7f4e70f6c000 r--s 00009000 fc:00 786503                     /home/q/Consolidator/lib/slf4j-api-1.7.21.jar
7f4e70f6c000-7f4e70f70000 r--s 00042000 fc:00 786504                     /home/q/Consolidator/lib/spray-json_2.11-1.3.2.jar
7f4e70f70000-7f4e70f7d000 r--s 000e2000 fc:00 786467                     /home/q/Consolidator/lib/akka-parsing_2.11-2.4.11.jar
7f4e70f7d000-7f4e70fb6000 r--s 00308000 fc:00 786464                     /home/q/Consolidator/lib/akka-http-core_2.11-2.4.11.jar
7f4e70fb6000-7f4e70ff5000 r--s 00256000 fc:00 786465                     /home/q/Consolidator/lib/akka-http-experimental_2.11-2.4.11.jar
7f4e70ff5000-7f4e70ffa000 r--s 00040000 fc:00 786472                     /home/q/Consolidator/lib/akka-testkit_2.11-2.4.14.jar
7f4e70ffa000-7f4e71000000 r--s 00039000 fc:00 786462                     /home/q/Consolidator/lib/Agrona-0.9.0.jar
7f4e71000000-7f4e71003000 r--s 00014000 fc:00 786460                     /home/q/Consolidator/lib/aeron-client-1.0.4.jar
7f4e71003000-7f4e7101c000 r--s 00123000 fc:00 786495                     /home/q/Consolidator/lib/netty-3.10.6.Final.jar
7f4e7101c000-7f4e71023000 r--s 00070000 fc:00 786468                     /home/q/Consolidator/lib/akka-protobuf_2.11-2.4.14.jar
7f4e71023000-7f4e7102e000 r--s 0005d000 fc:00 786501                     /home/q/Consolidator/lib/scala-parser-combinators_2.11-1.0.4.jar
7f4e7102e000-7f4e71079000 r--s 003e1000 fc:00 786471                     /home/q/Consolidator/lib/akka-stream_2.11-2.4.14.jar
7f4e71079000-7f4e7109f000 r--s 0023f000 fc:00 786469                     /home/q/Consolidator/lib/akka-remote_2.11-2.4.14.jar
7f4e7109f000-7f4e710ad000 r--s 00081000 fc:00 786497                     /home/q/Consolidator/lib/scala-java8-compat_2.11-0.7.0.jar
7f4e710ad000-7f4e710dc000 r--s 002f8000 fc:00 786463                     /home/q/Consolidator/lib/akka-actor_2.11-2.4.14.jar
7f4e710dc000-7f4e71134000 r--s 00523000 fc:00 786499                     /home/q/Consolidator/lib/scala-library-2.11.7.jar
7f4e71134000-7f4e71137000 ---p 00000000 00:00 0 
7f4e71137000-7f4e71235000 rw-p 00000000 00:00 0                          [stack:1942]
7f4e71235000-7f4e71238000 ---p 00000000 00:00 0 
7f4e71238000-7f4e71336000 rw-p 00000000 00:00 0                          [stack:1941]
7f4e71336000-7f4e71339000 ---p 00000000 00:00 0 
7f4e71339000-7f4e71437000 rw-p 00000000 00:00 0                          [stack:1940]
7f4e71437000-7f4e71438000 ---p 00000000 00:00 0 
7f4e71438000-7f4e72447000 rw-p 00000000 00:00 0                          [stack:1939]
7f4e72447000-7f4e72620000 r--s 03d3e000 fc:00 396259                     /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar
7f4e72620000-7f4e74eff000 rw-p 00000000 00:00 0 
7f4e74eff000-7f4e74f00000 ---p 00000000 00:00 0 
7f4e74f00000-7f4e75000000 rw-p 00000000 00:00 0                          [stack:1938]
7f4e75000000-7f4e76060000 rwxp 00000000 00:00 0 
7f4e76060000-7f4e84000000 ---p 00000000 00:00 0 
7f4e84000000-7f4e85029000 rw-p 00000000 00:00 0 
7f4e85029000-7f4e88000000 ---p 00000000 00:00 0 
7f4e88000000-7f4e88001000 r--s 00004000 fc:00 786470                     /home/q/Consolidator/lib/akka-slf4j_2.11-2.4.14.jar
7f4e88001000-7f4e88009000 r--s 00052000 fc:00 786505                     /home/q/Consolidator/lib/ssl-config-core_2.11-0.2.1.jar
7f4e88009000-7f4e88058000 rw-p 00000000 00:00 0 
7f4e88058000-7f4e88059000 ---p 00000000 00:00 0 
7f4e88059000-7f4e88159000 rw-p 00000000 00:00 0                          [stack:1937]
7f4e88159000-7f4e8815a000 ---p 00000000 00:00 0 
7f4e8815a000-7f4e8825a000 rw-p 00000000 00:00 0                          [stack:1936]
7f4e8825a000-7f4e8825b000 ---p 00000000 00:00 0 
7f4e8825b000-7f4e88385000 rw-p 00000000 00:00 0                          [stack:1935]
7f4e88385000-7f4e884a5000 ---p 00000000 00:00 0 
7f4e884a5000-7f4e884cf000 rw-p 00000000 00:00 0 
7f4e884cf000-7f4e885ef000 ---p 00000000 00:00 0 
7f4e885ef000-7f4e8868d000 rw-p 00000000 00:00 0 
7f4e8868d000-7f4e88694000 ---p 00000000 00:00 0 
7f4e88694000-7f4e886d7000 rw-p 00000000 00:00 0 
7f4e886d7000-7f4e88a55000 ---p 00000000 00:00 0 
7f4e88a55000-7f4e88a6f000 r-xp 00000000 fc:00 529988                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libzip.so
7f4e88a6f000-7f4e88c6f000 ---p 0001a000 fc:00 529988                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libzip.so
7f4e88c6f000-7f4e88c70000 rw-p 0001a000 fc:00 529988                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libzip.so
7f4e88c70000-7f4e88c7b000 r-xp 00000000 fc:00 1573091                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f4e88c7b000-7f4e88e7a000 ---p 0000b000 fc:00 1573091                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f4e88e7a000-7f4e88e7b000 r--p 0000a000 fc:00 1573091                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f4e88e7b000-7f4e88e7c000 rw-p 0000b000 fc:00 1573091                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f4e88e7c000-7f4e88e87000 r-xp 00000000 fc:00 1573095                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f4e88e87000-7f4e89086000 ---p 0000b000 fc:00 1573095                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f4e89086000-7f4e89087000 r--p 0000a000 fc:00 1573095                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f4e89087000-7f4e89088000 rw-p 0000b000 fc:00 1573095                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f4e89088000-7f4e8909f000 r-xp 00000000 fc:00 1573085                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7f4e8909f000-7f4e8929e000 ---p 00017000 fc:00 1573085                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7f4e8929e000-7f4e8929f000 r--p 00016000 fc:00 1573085                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7f4e8929f000-7f4e892a0000 rw-p 00017000 fc:00 1573085                    /lib/x86_64-linux-gnu/libnsl-2.19.so
7f4e892a0000-7f4e892a2000 rw-p 00000000 00:00 0 
7f4e892a2000-7f4e892ab000 r-xp 00000000 fc:00 1573087                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f4e892ab000-7f4e894aa000 ---p 00009000 fc:00 1573087                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f4e894aa000-7f4e894ab000 r--p 00008000 fc:00 1573087                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f4e894ab000-7f4e894ac000 rw-p 00009000 fc:00 1573087                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f4e894ac000-7f4e894b6000 r-xp 00000000 fc:00 529990                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libinstrument.so
7f4e894b6000-7f4e896b5000 ---p 0000a000 fc:00 529990                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libinstrument.so
7f4e896b5000-7f4e896b6000 rw-p 00009000 fc:00 529990                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libinstrument.so
7f4e896b6000-7f4e896e1000 r-xp 00000000 fc:00 530021                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
7f4e896e1000-7f4e898e0000 ---p 0002b000 fc:00 530021                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
7f4e898e0000-7f4e898e2000 rw-p 0002a000 fc:00 530021                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libjava.so
7f4e898e2000-7f4e898ef000 r-xp 00000000 fc:00 529999                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libverify.so
7f4e898ef000-7f4e89aee000 ---p 0000d000 fc:00 529999                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libverify.so
7f4e89aee000-7f4e89af0000 rw-p 0000c000 fc:00 529999                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/libverify.so
7f4e89af0000-7f4e89af7000 r-xp 00000000 fc:00 1573128                    /lib/x86_64-linux-gnu/librt-2.19.so
7f4e89af7000-7f4e89cf6000 ---p 00007000 fc:00 1573128                    /lib/x86_64-linux-gnu/librt-2.19.so
7f4e89cf6000-7f4e89cf7000 r--p 00006000 fc:00 1573128                    /lib/x86_64-linux-gnu/librt-2.19.so
7f4e89cf7000-7f4e89cf8000 rw-p 00007000 fc:00 1573128                    /lib/x86_64-linux-gnu/librt-2.19.so
7f4e89cf8000-7f4e89dfd000 r-xp 00000000 fc:00 1573070                    /lib/x86_64-linux-gnu/libm-2.19.so
7f4e89dfd000-7f4e89ffc000 ---p 00105000 fc:00 1573070                    /lib/x86_64-linux-gnu/libm-2.19.so
7f4e89ffc000-7f4e89ffd000 r--p 00104000 fc:00 1573070                    /lib/x86_64-linux-gnu/libm-2.19.so
7f4e89ffd000-7f4e89ffe000 rw-p 00105000 fc:00 1573070                    /lib/x86_64-linux-gnu/libm-2.19.so
7f4e89ffe000-7f4e8accc000 r-xp 00000000 fc:00 530002                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
7f4e8accc000-7f4e8aecc000 ---p 00cce000 fc:00 530002                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
7f4e8aecc000-7f4e8afa5000 rw-p 00cce000 fc:00 530002                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
7f4e8afa5000-7f4e8aff0000 rw-p 00000000 00:00 0 
7f4e8aff0000-7f4e8b1ac000 r-xp 00000000 fc:00 1573032                    /lib/x86_64-linux-gnu/libc-2.19.so
7f4e8b1ac000-7f4e8b3ab000 ---p 001bc000 fc:00 1573032                    /lib/x86_64-linux-gnu/libc-2.19.so
7f4e8b3ab000-7f4e8b3af000 r--p 001bb000 fc:00 1573032                    /lib/x86_64-linux-gnu/libc-2.19.so
7f4e8b3af000-7f4e8b3b1000 rw-p 001bf000 fc:00 1573032                    /lib/x86_64-linux-gnu/libc-2.19.so
7f4e8b3b1000-7f4e8b3b6000 rw-p 00000000 00:00 0 
7f4e8b3b6000-7f4e8b3b9000 r-xp 00000000 fc:00 1573048                    /lib/x86_64-linux-gnu/libdl-2.19.so
7f4e8b3b9000-7f4e8b5b8000 ---p 00003000 fc:00 1573048                    /lib/x86_64-linux-gnu/libdl-2.19.so
7f4e8b5b8000-7f4e8b5b9000 r--p 00002000 fc:00 1573048                    /lib/x86_64-linux-gnu/libdl-2.19.so
7f4e8b5b9000-7f4e8b5ba000 rw-p 00003000 fc:00 1573048                    /lib/x86_64-linux-gnu/libdl-2.19.so
7f4e8b5ba000-7f4e8b5cf000 r-xp 00000000 fc:00 529962                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/jli/libjli.so
7f4e8b5cf000-7f4e8b7cf000 ---p 00015000 fc:00 529962                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/jli/libjli.so
7f4e8b7cf000-7f4e8b7d0000 rw-p 00015000 fc:00 529962                     /usr/lib/jvm/java-8-oracle/jre/lib/amd64/jli/libjli.so
7f4e8b7d0000-7f4e8b7e9000 r-xp 00000000 fc:00 1573122                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7f4e8b7e9000-7f4e8b9e8000 ---p 00019000 fc:00 1573122                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7f4e8b9e8000-7f4e8b9e9000 r--p 00018000 fc:00 1573122                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7f4e8b9e9000-7f4e8b9ea000 rw-p 00019000 fc:00 1573122                    /lib/x86_64-linux-gnu/libpthread-2.19.so
7f4e8b9ea000-7f4e8b9ee000 rw-p 00000000 00:00 0 
7f4e8b9ee000-7f4e8ba11000 r-xp 00000000 fc:00 1573012                    /lib/x86_64-linux-gnu/ld-2.19.so
7f4e8ba11000-7f4e8ba13000 r--s 00005000 fc:00 786466                     /home/q/Consolidator/lib/akka-http-spray-json-experimental_2.11-2.4.11.jar
7f4e8ba13000-7f4e8ba17000 r--s 00029000 fc:00 786461                     /home/q/Consolidator/lib/aeron-driver-1.0.4.jar
7f4e8ba17000-7f4e8bafc000 rw-p 00000000 00:00 0 
7f4e8bafc000-7f4e8bb04000 rw-s 00000000 fc:00 2097162                    /tmp/hsperfdata_root/1929
7f4e8bb04000-7f4e8bb07000 ---p 00000000 00:00 0 
7f4e8bb07000-7f4e8bc09000 rw-p 00000000 00:00 0                          [stack:1934]
7f4e8bc09000-7f4e8bc0b000 r--s 0000a000 fc:00 786506                     /home/q/Consolidator/lib/uncommons-maths-1.2.2a.jar
7f4e8bc0b000-7f4e8bc0c000 r--s 00000000 fc:00 786496                     /home/q/Consolidator/lib/reactive-streams-1.0.0.jar
7f4e8bc0c000-7f4e8bc0d000 rw-p 00000000 00:00 0 
7f4e8bc0d000-7f4e8bc0e000 r--p 00000000 00:00 0 
7f4e8bc0e000-7f4e8bc10000 rw-p 00000000 00:00 0 
7f4e8bc10000-7f4e8bc11000 r--p 00022000 fc:00 1573012                    /lib/x86_64-linux-gnu/ld-2.19.so
7f4e8bc11000-7f4e8bc12000 rw-p 00023000 fc:00 1573012                    /lib/x86_64-linux-gnu/ld-2.19.so
7f4e8bc12000-7f4e8bc13000 rw-p 00000000 00:00 0 
7fff3a803000-7fff3a824000 rw-p 00000000 00:00 0                          [stack]
7fff3a9d7000-7fff3a9d9000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Dprofile.default.audit.docType=DictChange -javaagent:../lib/aspectjweaver-1.8.9.jar -Dapp.name=jobsConsolidator -Dapp.pid=1929 -Dapp.repo=/home/q/Consolidator/lib -Dapp.home=/home/q/Consolidator -Dbasedir=/home/q/Consolidator 
java_command: ro.softnrg.manny.consolidator.Application -p default
java_class_path (initial): /home/q/Consolidator/etc:/home/q/Consolidator/lib/scala-library-2.11.7.jar:/home/q/Consolidator/lib/akka-actor_2.11-2.4.14.jar:/home/q/Consolidator/lib/scala-java8-compat_2.11-0.7.0.jar:/home/q/Consolidator/lib/akka-remote_2.11-2.4.14.jar:/home/q/Consolidator/lib/akka-stream_2.11-2.4.14.jar:/home/q/Consolidator/lib/reactive-streams-1.0.0.jar:/home/q/Consolidator/lib/ssl-config-core_2.11-0.2.1.jar:/home/q/Consolidator/lib/scala-parser-combinators_2.11-1.0.4.jar:/home/q/Consolidator/lib/akka-protobuf_2.11-2.4.14.jar:/home/q/Consolidator/lib/netty-3.10.6.Final.jar:/home/q/Consolidator/lib/uncommons-maths-1.2.2a.jar:/home/q/Consolidator/lib/aeron-driver-1.0.4.jar:/home/q/Consolidator/lib/aeron-client-1.0.4.jar:/home/q/Consolidator/lib/Agrona-0.9.0.jar:/home/q/Consolidator/lib/akka-testkit_2.11-2.4.14.jar:/home/q/Consolidator/lib/akka-http-spray-json-experimental_2.11-2.4.11.jar:/home/q/Consolidator/lib/akka-http-experimental_2.11-2.4.11.jar:/home/q/Consolidator/lib/akka-http-core_2.11-2.4.11.jar:/home/q/Consolidator/lib/akka-parsing_2.11-2.4.11.jar:/home/q/Consolidator/lib/spray-json_2.11-1.3.2.jar:/home/q/Consolidator/lib/akka-slf4j_2.11-2.4.14.jar:/home/q/Consolidator/lib/slf4j-api-1.7.21.jar:/home/q/Consolidator/lib/log4j-slf4j-impl-2.7.jar:/home/q/Consolidator/lib/log4j-api-2.7.jar:/home/q/Consolidator/lib/log4j-core-2.7.jar:/home/q/Consolidator/lib/log4j-1.2-api-2.7.jar:/home/q/Consolidator/lib/config-1.3.0.jar:/home/q/Consolidator/lib/kamon-core_2.11-0.6.3.jar:/home/q/Consolidator/lib/HdrHistogram-2.1.8.jar:/home/q/Consolidator/lib/kamon-system-metrics_2.11-0.6.3.jar:/home/q/Consolidator/lib/sigar-loader-1.6.5-rev002.jar:/home/q/Consolidator/lib/kamon-akka_2.11-0.6.3.jar:/home/q/Consolidator/lib/kamon-scala_2.11-0.6.3.jar:/home/q/Consolidator/lib/aspectjweaver-1.8.9.jar:/home/q/Consolidator/lib/scala-json-0.2-SNAPSHOT.jar:/home/q/Consolidator/lib/scala-logger-0.2-SNAPSHOT.jar:/home/q/Consolidator/lib/jackson-databind-2.8.4.jar:/home/q/
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin

Signal Handlers:
SIGSEGV: [libjvm.so+0xac78c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xac78c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x920ac0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x920ac0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x920ac0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x920ac0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x922300], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO

---------------  S Y S T E M  ---------------

OS:DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

uname:Linux 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64
libc:glibc 2.19 NPTL 2.19 
rlimit: STACK 8192k, CORE 0k, NPROC 31449, NOFILE 1000000, AS infinity
load average:0.22 0.14 0.09

/proc/meminfo:
MemTotal:        4048016 kB
MemFree:         2866548 kB
Buffers:           18688 kB
Cached:           179780 kB
SwapCached:            0 kB
Active:           990636 kB
Inactive:         122416 kB
Active(anon):     914804 kB
Inactive(anon):      176 kB
Active(file):      75832 kB
Inactive(file):   122240 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4190204 kB
SwapFree:        4190204 kB
Dirty:                84 kB
Writeback:             0 kB
AnonPages:        914728 kB
Mapped:            22376 kB
Shmem:               400 kB
Slab:              24504 kB
SReclaimable:      13112 kB
SUnreclaim:        11392 kB
KernelStack:        1360 kB
PageTables:         5480 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6214212 kB
Committed_AS:    1222272 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       12080 kB
VmallocChunk:   34359714692 kB
HardwareCorrupted:     0 kB
AnonHugePages:    495616 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       38848 kB
DirectMap2M:     4155392 kB

CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, lzcnt, tsc, tscinvbit

/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
stepping    : 2
cpu MHz     : 3499.994
cache size  : 15360 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm
bogomips    : 6999.98
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
stepping    : 2
cpu MHz     : 3499.994
cache size  : 15360 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 4
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm
bogomips    : 6999.98
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
stepping    : 2
cpu MHz     : 3499.994
cache size  : 15360 KB
physical id : 0
siblings    : 4
core id     : 2
cpu cores   : 4
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm
bogomips    : 6999.98
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 63
model name  : Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz
stepping    : 2
cpu MHz     : 3499.994
cache size  : 15360 KB
physical id : 0
siblings    : 4
core id     : 3
cpu cores   : 4
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 15
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm
bogomips    : 6999.98
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Memory: 4k page, physical 4048016k(2866548k free), swap 4190204k(4190204k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Wed Mar  1 11:28:08 2017
elapsed time: 99 seconds (0d 0h 1m 39s)

Thanks Chris

ivantopo commented 7 years ago

I'm pretty sure that this is your issue: https://github.com/kamon-io/sigar-loader/tree/master/core/src/main/resources#provision-from-several-jvm

Please ensure that each JVM gets its own folder for unpacking the sigar native binaries and all should be good!

kcris commented 7 years ago

thanks a lot! I now pass 

-javaagent:../lib/sigar-loader-1.6.5-rev002.jar=kamon.sigar.folder=../tmp/sigar/x

to jvm, but I get a warning, kamon-akka and kamon-scala complain that they still want aspectj as javaagent (I used aspectj previously, I just switched now to sigar-loader due to my issue)

It seems like your application was not started with the -javaagent:/path-to-aspectj-weaver.jar option but Kamon detected
 the following modules which require AspectJ to work properly:

      kamon-akka, kamon-scala

what is the right way to deal with this?

thanks a lot Chris

ivantopo commented 7 years ago

Just ensure that your application is started with the -javaagent:/path-to-aspectjweaver.jar option and that's it, that's for the instrumentation provided by kamon-scala and kamon-akka to work properly.

kcris commented 7 years ago

so, I guess I have to pass both as javaagents -aspectj -sigar-loader (pointed by your link above), which allows customizing the kamon.sigar.folder

thanks a lot