paoloantinori
commented
6 years ago Is this still valid with the current version? A performance improvement should have improved the reported behavior.
Closed maschmid closed 6 years ago
paoloantinori
commented
6 years ago Is this still valid with the current version? A performance improvement should have improved the reported behavior.
heiko-braun
commented
6 years ago Is this related to #2579 ?
zregvart
commented
6 years ago This holds true for latest version of master. On an integration I've run for < 24H, from 11:11:38, the youngest activity seen is at 02:10:02 (15H later). The pod has been running without restarts.
heiko-braun
commented
6 years ago Good work @zregvart . Do we have an idea what might be causing this?
paoloantinori
commented
6 years ago a hypothesis could be a hanging query, but this should be easy to spot with either high load on a process at postgres level or a busy thread in a threaddump. probably too easy for us to be this lucky.
Are you noticing anything like that @zregvart ?
zregvart
commented
6 years ago What can be seen from the thread dumps is that we're reading logs from 13 threads -- this corresponds to 13 running integrations on staging.
Java thread dump:
Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):
"HystrixTimer-4" #85 daemon prio=5 os_prio=0 tid=0x00007f4d745a62f0 nid=0xc2 waiting on condition [0x00007f4d84a49000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd82368> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"hystrix-Meta-3" #84 daemon prio=5 os_prio=0 tid=0x00007f4de406a2e0 nid=0xc1 waiting on condition [0x00007f4d84b4a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd81200> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"HystrixTimer-3" #83 daemon prio=5 os_prio=0 tid=0x00007f4de4069090 nid=0xc0 waiting on condition [0x00007f4d84c4b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd82368> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"hystrix-Meta-2" #82 daemon prio=5 os_prio=0 tid=0x00007f4dc4006ea0 nid=0xbf waiting on condition [0x00007f4d84d4c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd81200> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"HystrixTimer-2" #81 daemon prio=5 os_prio=0 tid=0x00007f4dc40076e0 nid=0xbe waiting on condition [0x00007f4d84e4d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd82368> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"hystrix-Meta-1" #80 daemon prio=5 os_prio=0 tid=0x00000000017fe560 nid=0xbd waiting on condition [0x00007f4d84f4e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd81200> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"HystrixTimer-1" #79 daemon prio=5 os_prio=0 tid=0x000000000166fa70 nid=0xbc waiting on condition [0x00007f4d8504f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd82368> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxComputationScheduler-1" #78 daemon prio=5 os_prio=0 tid=0x00000000020d03b0 nid=0xbb waiting on condition [0x00007f4d85350000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd56bd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxIoScheduler-1 (Evictor)" #77 daemon prio=5 os_prio=0 tid=0x0000000001fd8410 nid=0xba waiting on condition [0x00007f4d85451000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd56de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxScheduledExecutorPool-2" #76 daemon prio=5 os_prio=0 tid=0x0000000002008890 nid=0xb9 waiting on condition [0x00007f4d86ff4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd57448> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RxScheduledExecutorPool-1" #75 daemon prio=5 os_prio=0 tid=0x00000000015bae50 nid=0xb8 waiting on condition [0x00007f4d86ef3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ebd57c80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-10" #74 prio=5 os_prio=0 tid=0x00007f4d7c254dc0 nid=0xb7 waiting on condition [0x00007f4d871f6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-9" #73 prio=5 os_prio=0 tid=0x00007f4d7c254980 nid=0xb6 waiting on condition [0x00007f4d86cb8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-8" #72 prio=5 os_prio=0 tid=0x00007f4d7c254540 nid=0xb5 waiting on condition [0x00007f4d86bb7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-7" #71 prio=5 os_prio=0 tid=0x00007f4d7c2642e0 nid=0xb4 waiting on condition [0x00007f4d85952000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-6" #70 prio=5 os_prio=0 tid=0x00007f4d7c2633c0 nid=0xb3 waiting on condition [0x00007f4d85a53000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-5" #69 prio=5 os_prio=0 tid=0x00007f4d7c312b90 nid=0xb2 waiting on condition [0x00007f4d85b54000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-10" #68 prio=5 os_prio=0 tid=0x0000000001671850 nid=0xb1 waiting on condition [0x00007f4d85c55000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-9" #67 prio=5 os_prio=0 tid=0x0000000001673ad0 nid=0xb0 waiting on condition [0x00007f4d85d56000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-4" #66 prio=5 os_prio=0 tid=0x00007f4d7c288470 nid=0xaf waiting on condition [0x00007f4d85e57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-8" #65 prio=5 os_prio=0 tid=0x00000000015c4c10 nid=0xae waiting on condition [0x00007f4d85f58000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-7" #64 prio=5 os_prio=0 tid=0x00000000015cc210 nid=0xad waiting on condition [0x00007f4d86059000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-3" #63 prio=5 os_prio=0 tid=0x00007f4d7c287410 nid=0xac waiting on condition [0x00007f4d8615a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-6" #62 prio=5 os_prio=0 tid=0x00000000015cb080 nid=0xab waiting on condition [0x00007f4d8645b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-5" #61 prio=5 os_prio=0 tid=0x00000000015bea90 nid=0xaa waiting on condition [0x00007f4d8655c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-2" #60 prio=5 os_prio=0 tid=0x00007f4d7c286fd0 nid=0xa9 waiting on condition [0x00007f4d8665d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-4" #59 prio=5 os_prio=0 tid=0x00000000015bdb70 nid=0xa8 waiting on condition [0x00007f4d8675e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-3" #58 prio=5 os_prio=0 tid=0x00000000017cc210 nid=0xa7 waiting on condition [0x00007f4d8685f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #57 prio=5 os_prio=0 tid=0x00007f4d940166d0 nid=0xa6 runnable [0x00007f4d86898000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb731120> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb7311e0> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #56 prio=5 os_prio=0 tid=0x00007f4db81b9ae0 nid=0xa5 runnable [0x00007f4d868d1000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb70ca18> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb70ca28> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #55 prio=5 os_prio=0 tid=0x00007f4db0001070 nid=0xa4 runnable [0x00007f4d86909000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb765550> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb766090> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #54 prio=5 os_prio=0 tid=0x00007f4d88057c40 nid=0xa3 runnable [0x00007f4d86943000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb6defb0> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb6e7bc0> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #53 prio=5 os_prio=0 tid=0x00007f4da0005110 nid=0xa2 runnable [0x00007f4d8697c000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb7356a0> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb735760> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #52 prio=5 os_prio=0 tid=0x00007f4d98001140 nid=0xa1 runnable [0x00007f4d869b4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb721740> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb724820> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-2" #51 prio=5 os_prio=0 tid=0x00000000017cbdd0 nid=0xa0 waiting on condition [0x00007f4d86ab6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #48 prio=5 os_prio=0 tid=0x00007f4d880918c0 nid=0x9d runnable [0x00007f4d86cf1000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb6d2778> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb6d2f28> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"OkHttp ConnectionPool" #47 daemon prio=5 os_prio=0 tid=0x00007f4d94000f30 nid=0x9c in Object.wait() [0x00007f4d86df2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
- locked <0x00000000e949c830> (a okhttp3.ConnectionPool)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #45 prio=5 os_prio=0 tid=0x00007f4d88011cf0 nid=0x9a runnable [0x00007f4dbc042000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb6d47e8> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb6dd3f8> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #43 prio=5 os_prio=0 tid=0x00007f4d88010a00 nid=0x98 runnable [0x00007f4dbc98b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb75eba8> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb75f560> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #42 prio=5 os_prio=0 tid=0x00007f4d880f51a0 nid=0x97 runnable [0x00007f4dbc9c4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb733358> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb7353d8> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"OkHttp Dispatcher" #41 prio=5 os_prio=0 tid=0x00007f4d8c0019b0 nid=0x96 waiting on condition [0x00007f4d870f5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb69ebb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #39 prio=5 os_prio=0 tid=0x00007f4d880f3bd0 nid=0x94 runnable [0x00007f4dbc9fd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb715060> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb7170d8> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #38 prio=5 os_prio=0 tid=0x00007f4d88064750 nid=0x93 runnable [0x00007f4de803b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000000eb74a130> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000000eb74aae8> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:52)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$onResponse$0(KubernetesSupport.java:84)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$201/103812548.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 task-1" #37 prio=5 os_prio=0 tid=0x00000000015c7fc0 nid=0x92 waiting on condition [0x00007f4d874f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb00bc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 task-1" #36 prio=5 os_prio=0 tid=0x00007f4d7c369190 nid=0x91 waiting on condition [0x00007f4d875f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eac14188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"DestroyJavaVM" #35 prio=5 os_prio=0 tid=0x00007f4e08008c20 nid=0x6b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ResourceUpdateController (scheduler)" #34 prio=5 os_prio=0 tid=0x00007f4e096019d0 nid=0x90 waiting on condition [0x00007f4d878f9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eb002240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-3 Accept" #32 prio=5 os_prio=0 tid=0x00007f4e092c9730 nid=0x8f runnable [0x00007f4d879fa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eb00eda0> (a sun.nio.ch.Util$3)
- locked <0x00000000eb00ed90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eb00b8a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"XNIO-3 I/O-2" #31 prio=5 os_prio=0 tid=0x00007f4e085ecad0 nid=0x8e runnable [0x00007f4d87afb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eb00c238> (a sun.nio.ch.Util$3)
- locked <0x00000000eb00c228> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eb00c110> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:524)
"XNIO-3 I/O-1" #30 prio=5 os_prio=0 tid=0x00007f4e09c91830 nid=0x8d runnable [0x00007f4d87bfc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eb00e168> (a sun.nio.ch.Util$3)
- locked <0x00000000eb00e158> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eb00e040> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"XNIO-2 Accept" #29 prio=5 os_prio=0 tid=0x00007f4e09c956c0 nid=0x8c runnable [0x00007f4d87cfd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eac37270> (a sun.nio.ch.Util$3)
- locked <0x00000000eac37260> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eac15358> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"XNIO-2 I/O-2" #28 prio=5 os_prio=0 tid=0x00007f4e09c95030 nid=0x8b runnable [0x00007f4d87dfe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eac14790> (a sun.nio.ch.Util$3)
- locked <0x00000000eac14780> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eac14668> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"XNIO-2 I/O-1" #27 prio=5 os_prio=0 tid=0x00007f4e092520b0 nid=0x8a runnable [0x00007f4db4122000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000eac14e10> (a sun.nio.ch.Util$3)
- locked <0x00000000eac14e00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000eac14ce8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"spring.cloud.inetutils" #26 daemon prio=5 os_prio=0 tid=0x00007f4e09241f40 nid=0x89 waiting on condition [0x00007f4db4223000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ea792280> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"pool-6-thread-1" #25 prio=5 os_prio=0 tid=0x00007f4e092e1690 nid=0x88 waiting on condition [0x00007f4db4724000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e96bed30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller Scheduler" #24 prio=5 os_prio=0 tid=0x00007f4e092e1fa0 nid=0x87 waiting on condition [0x00007f4de8074000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e949cd60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Logs Controller" #23 prio=5 os_prio=0 tid=0x00007f4e092e1b60 nid=0x86 waiting on condition [0x00007f4df805a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e949d300> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
at io.syndesis.server.logging.jsondb.controller.ActivityTrackingController.processEventQueue(ActivityTrackingController.java:320)
at io.syndesis.server.logging.jsondb.controller.ActivityTrackingController$$Lambda$97/656898222.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Okio Watchdog" #22 daemon prio=5 os_prio=0 tid=0x00007f4da40db400 nid=0x85 in Object.wait() [0x00007f4db4c25000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:347)
at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
- locked <0x00000000e8c42360> (a java.lang.Class for okio.AsyncTimeout)
"OkHttp ConnectionPool" #21 daemon prio=5 os_prio=0 tid=0x00007f4da40e6840 nid=0x84 in Object.wait() [0x00007f4db4d26000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
- locked <0x00000000e8907690> (a okhttp3.ConnectionPool)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Integration Controller Scheduler" #20 prio=5 os_prio=0 tid=0x00007f4da0002490 nid=0x83 waiting on condition [0x00007f4db5027000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e896c4c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Integration Controller" #19 prio=5 os_prio=0 tid=0x00007f4da0001260 nid=0x82 waiting on condition [0x00007f4db5128000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e896beb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Integration Controller Scheduler" #18 prio=5 os_prio=0 tid=0x00007f4e0975e010 nid=0x81 waiting on condition [0x00007f4db5229000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e896c4c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Endpoint Controller" #17 prio=5 os_prio=0 tid=0x00007f4e095c4bf0 nid=0x80 waiting on condition [0x00007f4db532a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e89671c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"Tomcat JDBC Pool Cleaner[1706377736:1532374333218]" #16 daemon prio=5 os_prio=0 tid=0x00007f4e0942f810 nid=0x7f in Object.wait() [0x00007f4dbc143000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000e7ef1048> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"XNIO-1 Accept" #15 daemon prio=5 os_prio=0 tid=0x00007f4e082e3a30 nid=0x7e runnable [0x00007f4de8175000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e7c887f0> (a sun.nio.ch.Util$3)
- locked <0x00000000e7c887e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e7c886c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"XNIO-1 I/O-1" #14 daemon prio=5 os_prio=0 tid=0x00007f4e08ce9530 nid=0x7d runnable [0x00007f4de9080000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000e7c88170> (a sun.nio.ch.Util$3)
- locked <0x00000000e7c88160> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e7c88048> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:511)
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f4e08193a20 nid=0x77 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4e0818f1c0 nid=0x76 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4e0801a7f0 nid=0x75 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4e08180fe0 nid=0x74 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4e0817f530 nid=0x73 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4e08156cb0 nid=0x72 in Object.wait() [0x00007f4de999f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000e7006c78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4e08152a50 nid=0x71 in Object.wait() [0x00007f4de9aa0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e7004cb8> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"VM Thread" os_prio=0 tid=0x00007f4e08149550 nid=0x70 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4e0801d6b0 nid=0x6c runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4e0801ec40 nid=0x6d runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4e080201d0 nid=0x6e runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4e08021760 nid=0x6f runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f4e08195ac0 nid=0x78 waiting on condition
JNI global references: 788
Heap
PSYoungGen total 74240K, used 48476K [0x00000000f7b00000, 0x00000000fd100000, 0x0000000100000000)
eden space 60928K, 57% used [0x00000000f7b00000,0x00000000f9d7f978,0x00000000fb680000)
from space 13312K, 98% used [0x00000000fc400000,0x00000000fd0d7a58,0x00000000fd100000)
to space 13312K, 0% used [0x00000000fb700000,0x00000000fb700000,0x00000000fc400000)
ParOldGen total 183808K, used 113150K [0x00000000e7000000, 0x00000000f2380000, 0x00000000f7b00000)
object space 183808K, 61% used [0x00000000e7000000,0x00000000ede7fb60,0x00000000f2380000)
Metaspace used 84667K, capacity 88720K, committed 88832K, reserved 1128448K
class space used 9775K, capacity 10468K, committed 10496K, reserved 1048576KI don't see anything out of the ordinary in PostgreSQL activity:
syndesis=> SELECT * FROM pg_stat_activity;
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | waiting | state | backend_xid | backend_xmin | query
-------+----------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+---------------------+-------------+--------------+----------------------------------------------------------------------------------------------------------------
16385 | syndesis | 25524 | 16384 | syndesis | psql | | | -1 | 2018-07-13 10:01:35.810835+00 | | 2018-07-13 10:15:27.431474+00 | 2018-07-13 10:16:40.848726+00 | f | idle | | | select count(*) from jsondb where path like '/integration-deployments/%';
16385 | syndesis | 4520 | 16384 | syndesis | | 10.1.5.1 | | 46322 | 2018-07-23 19:32:13.278578+00 | | 2018-07-23 19:32:13.330445+00 | 2018-07-23 19:32:13.330475+00 | f | idle | | | SET extra_float_digits = 3
16385 | syndesis | 4521 | 16384 | syndesis | | 10.1.5.1 | | 46324 | 2018-07-23 19:32:13.349131+00 | | 2018-07-23 19:32:13.352202+00 | 2018-07-23 19:32:13.352221+00 | f | idle | | | SET extra_float_digits = 3
16385 | syndesis | 33423 | 16384 | syndesis | psql | | | -1 | 2018-07-13 10:48:12.197831+00 | | 2018-07-13 10:56:04.972467+00 | 2018-07-13 10:56:04.972977+00 | f | idle | | | explain SELECT path, VALUE, ovalue FROM jsondb WHERE path LIKE '/integration-deployments/%' ORDER BY path ASC;
16385 | syndesis | 4522 | 16384 | syndesis | | 10.1.5.1 | | 46326 | 2018-07-23 19:32:13.356549+00 | | 2018-07-23 19:32:13.359907+00 | 2018-07-23 19:32:13.359933+00 | f | idle | | | SET extra_float_digits = 3
16385 | syndesis | 4523 | 16384 | syndesis | | 10.1.5.1 | | 46328 | 2018-07-23 19:32:13.365513+00 | | 2018-07-23 19:32:13.368345+00 | 2018-07-23 19:32:13.368365+00 | f | idle | | | SET extra_float_digits = 3
16385 | syndesis | 4524 | 16384 | syndesis | | 10.1.5.1 | | 46330 | 2018-07-23 19:32:13.372565+00 | 2018-07-24 11:03:21.662113+00 | 2018-07-24 11:03:21.662544+00 | 2018-07-24 11:03:21.662597+00 | f | idle in transaction | | 4866321 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
16385 | syndesis | 4525 | 16384 | syndesis | | 10.1.5.1 | | 46332 | 2018-07-23 19:32:13.381535+00 | | 2018-07-23 21:54:18.880504+00 | 2018-07-23 21:54:18.880512+00 | f | idle | | | COMMIT
16385 | syndesis | 4526 | 16384 | syndesis | | 10.1.5.1 | | 46334 | 2018-07-23 19:32:13.390177+00 | 2018-07-24 10:14:23.418601+00 | 2018-07-24 10:14:23.418892+00 | 2018-07-24 10:14:23.463398+00 | f | idle in transaction | | 4866207 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
16385 | syndesis | 4527 | 16384 | syndesis | | 10.1.5.1 | | 46336 | 2018-07-23 19:32:13.39722+00 | | 2018-07-23 21:54:38.583857+00 | 2018-07-23 21:54:38.58518+00 | f | idle | | | COMMIT
16385 | syndesis | 4528 | 16384 | syndesis | | 10.1.5.1 | | 46338 | 2018-07-23 19:32:13.404437+00 | | 2018-07-24 11:05:23.582884+00 | 2018-07-24 11:05:23.582891+00 | f | idle | | | COMMIT
16385 | syndesis | 4529 | 16384 | syndesis | | 10.1.5.1 | | 46340 | 2018-07-23 19:32:13.414632+00 | 2018-07-24 11:05:27.513837+00 | 2018-07-24 11:05:27.514039+00 | 2018-07-24 11:05:27.514086+00 | f | idle in transaction | | 4866325 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
16386 | sampledb | 93878 | 16387 | sampledb | | | | | | | | | | | | | <insufficient privilege>
16386 | sampledb | 93880 | 16387 | sampledb | | | | | | | | | | | | | <insufficient privilege>
16385 | syndesis | 68113 | 16384 | syndesis | psql | | | -1 | 2018-07-24 11:05:02.968675+00 | 2018-07-24 11:05:28.249001+00 | 2018-07-24 11:05:28.249001+00 | 2018-07-24 11:05:28.249003+00 | f | active | | 4866325 | SELECT * FROM pg_stat_activity;
(15 rows)I isolated and reproduced this locally and I have a theory.
After 05:50:37.263 ActivityTrackingController stopped ingesting new output from the pod:
05:50:34.206 [Logs Controller [running]: processEventQueue] DEBUG io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Batch ingested 8 log events
05:50:35.654 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Purging old controller
05:50:35.679 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - deleted 60 transactions for integration: i-LI5Ob1A80WPlhtKRBewz
05:50:36.119 [Logs Controller [running]: processEventQueue] DEBUG io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Batch ingested 1 log events
05:50:37.263 [Logs Controller [running]: processEventQueue] DEBUG io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Batch ingested 1 log events
05:51:35.680 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Purging old controller
05:51:35.705 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - deleted 60 transactions for integration: i-LI5Ob1A80WPlhtKRBewz
05:52:35.706 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Purging old controller
05:52:35.732 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - deleted 60 transactions for integration: i-LI5Ob1A80WPlhtKRBewz
05:53:35.732 [Logs Controller Scheduler [running]: cleanupLogs] INFO io.syndesis.server.logging.jsondb.controller.ActivityTrackingController - Purging old controllerI see an established connection from the reproducer:
$ lsof -i4 -i6 -n|grep 24908
java 24908 zregvart 0u IPv6 2295175 0t0 TCP 127.0.0.1:45571->127.0.0.1:51560 (ESTABLISHED)
java 24908 zregvart 5u IPv6 2302434 0t0 TCP 127.0.0.1:45571->127.0.0.1:51562 (ESTABLISHED)
java 24908 zregvart 160u IPv6 1836753 0t0 TCP 192.168.1.74:50066->52.20.219.125:https (ESTABLISHED)
java 24908 zregvart 162u IPv6 2263101 0t0 TCP *:45571 (LISTEN)And the thread dump (I changed thread names to make it easier to troubleshoot):
"Logs Controller [running], streaming: https://api.rh-idev.openshift.com/api/v1/namespaces/syndesis-staging/pods/i-1sec-timer-to-log-1-qplzg/log?pretty=false&follow=true×tamps=true&sinceTime=" #16 prio=5 os_prio=0 tid=0x00007fc180142800 nid=0x617d runnable [0x00007fc1aaed7000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
- locked <0x00000006cea39e30> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000006cec99838> (a sun.security.ssl.AppInputStream)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.request(RealBufferedSource.java:67)
at okio.RealBufferedSource.require(RealBufferedSource.java:60)
at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:294)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.readChunkSize(Http1Codec.java:444)
at okhttp3.internal.http1.Http1Codec$ChunkedSource.read(Http1Codec.java:425)
at okio.RealBufferedSource$1.read(RealBufferedSource.java:419)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.processLogStream(PodLogMonitor.java:139)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:121)
at io.syndesis.server.logging.jsondb.controller.PodLogMonitor.accept(PodLogMonitor.java:1)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1.lambda$0(KubernetesSupport.java:90)
at io.syndesis.server.logging.jsondb.controller.KubernetesSupport$1$$Lambda$23/402131887.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000006cea38460> (a java.util.concurrent.ThreadPoolExecutor$Worker)This looks like a networking issue to me. For instance a firewall silently dropping a connection leaving our side half-open.
Also, when I started the ActivityTrackingController the IP address of api.rh-idev.openshift.com was resolved as 52.20.219.125, now the IP has changed to 52.44.190.13 and 52.2.249.192:
$ dig api.rh-idev.openshift.com ; <<>> DiG 9.11.4-RedHat-9.11.4-1.fc28 <<>> api.rh-idev.openshift.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25691
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.rh-idev.openshift.com. IN A
;; ANSWER SECTION:
api.rh-idev.openshift.com. 857 IN CNAME rh-idev-master-822484023.us-east-1.elb.amazonaws.com.
rh-idev-master-822484023.us-east-1.elb.amazonaws.com. 17 IN A 52.44.190.13
rh-idev-master-822484023.us-east-1.elb.amazonaws.com. 17 IN A 52.2.249.192
;; Query time: 2 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Wed Jul 25 08:06:03 CEST 2018
;; MSG SIZE rcvd: 152I think we need to rethink the long running pod log collection.
Starting with changing the read timeout from 0 here:
And handling the timeout and re-issuing the request. Not sure how to handle changing IPs though as Java caches DNS resolution, perhaps we should use some other library/tool to gather pod logs.
Also this could be only something that's happening outside of the cluster, but the symptoms look the same.
heiko-braun
commented
6 years ago Who engages the long running, persistent connection? Is it the log consumer polling for log entries?
zregvart
commented
6 years ago Who engages the long running, persistent connection? Is it the log consumer polling for log entries?
Yes from server-logging-jsondb module, here: https://github.com/syndesisio/syndesis/blob/8b825ec5cb78096db8819d7391002a829bb3abe1/app/server/logging/jsondb/src/main/java/io/syndesis/server/logging/jsondb/controller/KubernetesSupport.java#L61
This is a...
The problem
The activity log seems to stop at some point and does not show any of the recent entries. I have an integration that starts with a 1s timer, and after a day of runtime, the last activity shown is May 10, 2018, 7:01:44 PM
, while the current time is May 11, 2018 17:52:13 PM
watching the actual pod log shows that the integration is still running correctly and reports logs every seconds as expected.
Expected behavior
activity log should show recent entries even after days of uptime
Screenshot
Request and Response Data
API Endpoints and Schemas
Tasks involved / Steps to Reproduce
1. 2. 3. 4.