search

ktorio / ktor

Framework for quickly creating connected applications in Kotlin with minimal effort
https://ktor.io
Apache License 2.0
12.82k stars 1.04k forks source link

CPU usage at 100% Ktor HTTP Client CIO Engine #1041

Closed rocketraman closed 5 years ago

rocketraman commented 5 years ago

Ktor Version

1.1.3

Ktor Engine Used(client or server and name)

Client, CIO

JVM Version, Operating System and Relevant Context

OpenJDK 64-Bit Server VM (11.0.1+13-Debian-2 mixed mode, sharing) Debian buster/sid (openjdk:11 from https://hub.docker.com/_/openjdk/)

Feedback

Randomly, my process containing a ktor CIO client will show 100% CPU usage. Upon investigation of thread-level CPU usage, I can see that this ktor thread is in a tight loop:

"DefaultDispatcher-worker-5" #237 daemon prio=5 os_prio=0 cpu=238247952.22ms elapsed=238285.87s tid=0x00007f8aa4004800 nid=0x164 runnable  [0x00007f8acf19e000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:114)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a4e50c18> (a sun.nio.ch.Util$2)
    - locked <0x00000000a4e50ac8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.selectNow(java.base@11.0.1/SelectorImpl.java:146)
    at io.ktor.network.selector.ActorSelectorManager.process(ActorSelectorManager.kt:77)
    at io.ktor.network.selector.ActorSelectorManager$process$1.invokeSuspend(ActorSelectorManager.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)

Here is the full thread dump, for reference:

2019-03-25 14:48:25
Full thread dump OpenJDK 64-Bit Server VM (11.0.1+13-Debian-2 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f8b20001eb0, length=66, elements={
0x00007f8b68013800, 0x00007f8b680dd800, 0x00007f8b680df800, 0x00007f8b680e7000,
0x00007f8b680e9000, 0x00007f8b680eb000, 0x00007f8b680ed000, 0x00007f8b6810e000,
0x00007f8b6813d000, 0x00007f8b68ca9800, 0x00007f8b68cb0800, 0x00007f8b68cc6800,
0x00007f8b69574000, 0x00007f8af0001000, 0x00007f8af0010800, 0x00007f8af0012000,
0x00007f8af0014000, 0x00007f8b695c7000, 0x00007f8b697aa800, 0x00007f8b697b7800,
0x00007f8b697ba000, 0x00007f8b697bc000, 0x00007f8b697bf000, 0x00007f8b69c36000,
0x00007f8b69c78000, 0x00007f8b69c8b000, 0x00007f8b69c94000, 0x00007f8b6a313000,
0x00007f8b6a423000, 0x00007f8b68d30000, 0x00007f8b68d31800, 0x00007f8b68d33000,
0x00007f8b68d35800, 0x00007f8b69b1c000, 0x00007f8a8801c000, 0x00007f8a84002800,
0x00007f8a7c00c000, 0x00007f8a80005000, 0x00007f8b69715800, 0x00007f8a80010800,
0x00007f8a8000b000, 0x00007f8a80014000, 0x00007f8a6800a000, 0x00007f8a4800d000,
0x00007f8aa8016000, 0x00007f8a80302000, 0x00007f8a80ef4800, 0x00007f8a80f7f000,
0x00007f8a800d7800, 0x00007f8a80219800, 0x00007f8a8102b800, 0x00007f8a80fae000,
0x00007f8a8023f000, 0x00007f8a800ce000, 0x00007f8a80d7f000, 0x00007f8a803ab000,
0x00007f8a6c103000, 0x00007f8ab0001000, 0x00007f8aa4006800, 0x00007f8af400e000,
0x00007f8b2c00c800, 0x00007f8ab0008800, 0x00007f8b28004000, 0x00007f8aa4004800,
0x00007f8b28005800, 0x00007f8b20001000
}

"main" #1 prio=5 os_prio=0 cpu=5462.23ms elapsed=291218.58s tid=0x00007f8b68013800 nid=0x10 waiting on condition  [0x00007f8b6d796000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a260ca10> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@11.0.1/SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@11.0.1/SynchronousQueue.java:361)
    at java.util.concurrent.SynchronousQueue.take(java.base@11.0.1/SynchronousQueue.java:920)
    at com.xx.microservice.boot.Bootable.awaitTermination(Bootable.kt:72)
    at com.xx.microservice.boot.Bootable.boot$lib_boot(Bootable.kt:45)
    at com.xx.microservice.boot.BootableModuleKt.boot(BootableModule.kt:32)
    at com.xx.microservice.cis.AppKt.main(App.kt:12)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=17.66ms elapsed=291218.57s tid=0x00007f8b680dd800 nid=0x17 waiting on condition  [0x00007f8b50501000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.1/Native Method)
    at java.lang.ref.Reference.processPendingReferences(java.base@11.0.1/Reference.java:241)
    at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.1/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=3.45ms elapsed=291218.57s tid=0x00007f8b680df800 nid=0x18 in Object.wait()  [0x00007f8b50400000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:155)
    - waiting to re-lock in wait() <0x00000000a08e35f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:176)
    at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.1/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.93ms elapsed=291218.56s tid=0x00007f8b680e7000 nid=0x19 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=89211.76ms elapsed=291218.56s tid=0x00007f8b680e9000 nid=0x1a waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=17009.73ms elapsed=291218.56s tid=0x00007f8b680eb000 nid=0x1b waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #8 daemon prio=9 os_prio=0 cpu=3864.02ms elapsed=291218.56s tid=0x00007f8b680ed000 nid=0x1c runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #9 daemon prio=8 os_prio=0 cpu=160.70ms elapsed=291218.53s tid=0x00007f8b6810e000 nid=0x1d in Object.wait()  [0x00007f8b484fb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:155)
    - waiting to re-lock in wait() <0x00000000a0cdb460> (a java.lang.ref.ReferenceQueue$Lock)
    at jdk.internal.ref.CleanerImpl.run(java.base@11.0.1/CleanerImpl.java:148)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)
    at jdk.internal.misc.InnocuousThread.run(java.base@11.0.1/InnocuousThread.java:134)

"Service Thread" #10 daemon prio=9 os_prio=0 cpu=0.06ms elapsed=291218.50s tid=0x00007f8b6813d000 nid=0x1e runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RMI TCP Accept-0" #12 daemon prio=5 os_prio=0 cpu=0.44ms elapsed=291218.05s tid=0x00007f8b68ca9800 nid=0x21 runnable  [0x00007f8b41ffe000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(java.base@11.0.1/Native Method)
    at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.1/AbstractPlainSocketImpl.java:458)
    at java.net.ServerSocket.implAccept(java.base@11.0.1/ServerSocket.java:551)
    at java.net.ServerSocket.accept(java.base@11.0.1/ServerSocket.java:519)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(java.rmi@11.0.1/TCPTransport.java:394)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(java.rmi@11.0.1/TCPTransport.java:366)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"RMI TCP Accept-9010" #13 daemon prio=5 os_prio=0 cpu=0.13ms elapsed=291218.04s tid=0x00007f8b68cb0800 nid=0x22 runnable  [0x00007f8b41efd000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(java.base@11.0.1/Native Method)
    at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.1/AbstractPlainSocketImpl.java:458)
    at java.net.ServerSocket.implAccept(java.base@11.0.1/ServerSocket.java:551)
    at java.net.ServerSocket.accept(java.base@11.0.1/ServerSocket.java:519)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(java.rmi@11.0.1/TCPTransport.java:394)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(java.rmi@11.0.1/TCPTransport.java:366)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"RMI TCP Accept-0" #14 daemon prio=5 os_prio=0 cpu=0.23ms elapsed=291218.01s tid=0x00007f8b68cc6800 nid=0x23 runnable  [0x00007f8b41dfc000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(java.base@11.0.1/Native Method)
    at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.1/AbstractPlainSocketImpl.java:458)
    at java.net.ServerSocket.implAccept(java.base@11.0.1/ServerSocket.java:551)
    at java.net.ServerSocket.accept(java.base@11.0.1/ServerSocket.java:519)
    at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(jdk.management.agent@11.0.1/LocalRMIServerSocketFactory.java:52)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(java.rmi@11.0.1/TCPTransport.java:394)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(java.rmi@11.0.1/TCPTransport.java:366)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-2-thread-1" #16 prio=5 os_prio=0 cpu=5247.38ms elapsed=291216.66s tid=0x00007f8b69574000 nid=0x25 runnable  [0x00007f8b40df9000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a13002e0> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1300288> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"I/O dispatcher 1" #17 prio=5 os_prio=0 cpu=6036.89ms elapsed=291216.65s tid=0x00007f8af0001000 nid=0x26 runnable  [0x00007f8b40cf8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1315050> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1314ff8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"I/O dispatcher 2" #18 prio=5 os_prio=0 cpu=5656.06ms elapsed=291216.65s tid=0x00007f8af0010800 nid=0x27 runnable  [0x00007f8b40bf7000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a13153e8> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1315390> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"I/O dispatcher 3" #19 prio=5 os_prio=0 cpu=4993.28ms elapsed=291216.65s tid=0x00007f8af0012000 nid=0x28 runnable  [0x00007f8b40af6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1300738> (a sun.nio.ch.Util$2)
    - locked <0x00000000a13006e0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"I/O dispatcher 4" #20 prio=5 os_prio=0 cpu=5131.34ms elapsed=291216.65s tid=0x00007f8af0014000 nid=0x29 runnable  [0x00007f8b409f5000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1315630> (a sun.nio.ch.Util$2)
    - locked <0x00000000a13155d8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-3-thread-1" #21 prio=5 os_prio=0 cpu=182.77ms elapsed=291216.52s tid=0x00007f8b695c7000 nid=0x2a runnable  [0x00007f8b408f4000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-4-thread-1" #22 prio=5 os_prio=0 cpu=173.74ms elapsed=291216.48s tid=0x00007f8b697aa800 nid=0x2b runnable  [0x00007f8b407f3000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-5-thread-1" #23 prio=5 os_prio=0 cpu=190.34ms elapsed=291216.46s tid=0x00007f8b697b7800 nid=0x2c runnable  [0x00007f8b406f2000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-6-thread-1" #24 prio=5 os_prio=0 cpu=180.09ms elapsed=291216.46s tid=0x00007f8b697ba000 nid=0x2d runnable  [0x00007f8b405f1000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-7-thread-1" #25 prio=5 os_prio=0 cpu=181.49ms elapsed=291216.45s tid=0x00007f8b697bc000 nid=0x2e runnable  [0x00007f8b402d6000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-8-thread-1" #26 prio=5 os_prio=0 cpu=172.94ms elapsed=291216.45s tid=0x00007f8b697bf000 nid=0x2f runnable  [0x00007f8b401d5000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | producer-1" #27 daemon prio=5 os_prio=0 cpu=1075.64ms elapsed=291216.06s tid=0x00007f8b69c36000 nid=0x32 runnable  [0x00007f8acf9a3000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1b2df40> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1b2dee8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | producer-2" #28 daemon prio=5 os_prio=0 cpu=915.83ms elapsed=291215.94s tid=0x00007f8b69c78000 nid=0x33 runnable  [0x00007f8acf6a2000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1e8d380> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1e8d390> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | producer-3" #29 daemon prio=5 os_prio=0 cpu=923.12ms elapsed=291215.91s tid=0x00007f8b69c8b000 nid=0x34 runnable  [0x00007f8acf5a1000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1e8e550> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1e8e560> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | producer-4" #30 daemon prio=5 os_prio=0 cpu=915.24ms elapsed=291215.89s tid=0x00007f8b69c94000 nid=0x35 runnable  [0x00007f8acf4a0000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a1b2e248> (a sun.nio.ch.Util$2)
    - locked <0x00000000a1b2e1f0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-admin-client-thread | prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-admin" #37 daemon prio=5 os_prio=0 cpu=956.47ms elapsed=291213.35s tid=0x00007f8b6a313000 nid=0x43 runnable  [0x00007f8ace6e8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2983fd8> (a sun.nio.ch.Util$2)
    - locked <0x00000000a2983f80> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1125)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-9-thread-1" #40 prio=5 os_prio=0 cpu=178.33ms elapsed=291213.14s tid=0x00007f8b6a423000 nid=0x44 runnable  [0x00007f8ace7e9000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-consumer-prod-client-events" #41 daemon prio=5 os_prio=0 cpu=144151.01ms elapsed=291212.87s tid=0x00007f8b68d30000 nid=0x45 runnable  [0x00007f8acd0e1000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a51edf20> (a sun.nio.ch.Util$2)
    - locked <0x00000000a51eddd0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1243)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer.run(BasicCommitAfterProcessingConsumer.kt:48)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer$run$1.invokeSuspend(BasicCommitAfterProcessingConsumer.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.1/Executors.java:515)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.1/ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-consumer-prod-file-events" #42 daemon prio=5 os_prio=0 cpu=374856.95ms elapsed=291212.87s tid=0x00007f8b68d31800 nid=0x46 runnable  [0x00007f8accfe0000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a51560a0> (a sun.nio.ch.Util$2)
    - locked <0x00000000a5155f50> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1243)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer.run(BasicCommitAfterProcessingConsumer.kt:48)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer$run$1.invokeSuspend(BasicCommitAfterProcessingConsumer.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.1/Executors.java:515)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.1/ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-consumer-prod-segment-events" #43 daemon prio=5 os_prio=0 cpu=413459.47ms elapsed=291212.87s tid=0x00007f8b68d33000 nid=0x47 runnable  [0x00007f8accedf000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a5143bc8> (a sun.nio.ch.Util$2)
    - locked <0x00000000a5143a78> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1243)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer.run(BasicCommitAfterProcessingConsumer.kt:48)
    at com.xx.microservice.kafka.BasicCommitAfterProcessingConsumer$run$1.invokeSuspend(BasicCommitAfterProcessingConsumer.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.1/Executors.java:515)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.1/ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-StreamThread-1" #38 prio=5 os_prio=0 cpu=508563.67ms elapsed=291212.86s tid=0x00007f8b68d35800 nid=0x48 runnable  [0x00007f8accdde000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2989518> (a sun.nio.ch.Util$2)
    - locked <0x00000000a2989398> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1243)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1188)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1164)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:913)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:822)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)

"prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-CleanupThread" #44 daemon prio=5 os_prio=0 cpu=79.54ms elapsed=291212.86s tid=0x00007f8b69b1c000 nid=0x49 waiting on condition  [0x00007f8acccdd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a2984168> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.1/AbstractQueuedSynchronizer.java:2123)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-coordinator-heartbeat-thread | prod-cis" #45 daemon prio=5 os_prio=0 cpu=80322.96ms elapsed=291212.84s tid=0x00007f8a8801c000 nid=0x4a waiting on condition  [0x00007f8accbdc000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a51bfb48> (a java.util.concurrent.locks.ReentrantLock$FairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.1/AbstractQueuedSynchronizer.java:885)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.1/AbstractQueuedSynchronizer.java:917)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.1/AbstractQueuedSynchronizer.java:1240)
    at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.1/ReentrantLock.java:267)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:249)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:304)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:996)
    - locked <0x00000000a51519b0> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

"kafka-coordinator-heartbeat-thread | prod-cis" #46 daemon prio=5 os_prio=0 cpu=42320.78ms elapsed=291212.83s tid=0x00007f8a84002800 nid=0x4b waiting on condition  [0x00007f8accadb000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a51f4f48> (a java.util.concurrent.locks.ReentrantLock$FairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.1/AbstractQueuedSynchronizer.java:885)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.1/AbstractQueuedSynchronizer.java:917)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.1/AbstractQueuedSynchronizer.java:1240)
    at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.1/ReentrantLock.java:267)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:249)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:304)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:996)
    - locked <0x00000000a51ec5d0> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

"kafka-coordinator-heartbeat-thread | prod-cis" #47 daemon prio=5 os_prio=0 cpu=80145.12ms elapsed=291212.83s tid=0x00007f8a7c00c000 nid=0x4c waiting on condition  [0x00007f8acc9da000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a5146da8> (a java.util.concurrent.locks.ReentrantLock$FairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.1/AbstractQueuedSynchronizer.java:885)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.1/AbstractQueuedSynchronizer.java:917)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.1/AbstractQueuedSynchronizer.java:1240)
    at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.1/ReentrantLock.java:267)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:249)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:304)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:996)
    - locked <0x00000000a5142158> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

"kafka-coordinator-heartbeat-thread | prod-cisSegmenter" #48 daemon prio=5 os_prio=0 cpu=152643.89ms elapsed=291212.81s tid=0x00007f8a80005000 nid=0x4d in Object.wait()  [0x00007f8acc8d9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.1/Native Method)
    - waiting on <no object reference available>
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1021)
    - waiting to re-lock in wait() <0x00000000a2984310> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

"grpc-default-boss-ELG-1-1" #49 daemon prio=5 os_prio=0 cpu=6326.83ms elapsed=291212.72s tid=0x00007f8b69715800 nid=0x4e runnable  [0x00007f8acc5d8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a260d138> (a io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
    - locked <0x00000000a260d150> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-StreamThread-1-0_2-producer" #57 daemon prio=5 os_prio=0 cpu=1470.16ms elapsed=291123.81s tid=0x00007f8a80010800 nid=0x57 runnable  [0x00007f8acc3d6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2fdb120> (a sun.nio.ch.Util$2)
    - locked <0x00000000a2fdb130> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-StreamThread-1-0_6-producer" #58 daemon prio=5 os_prio=0 cpu=1361.12ms elapsed=291123.59s tid=0x00007f8a8000b000 nid=0x58 runnable  [0x00007f8acc4d7000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2fdb5f0> (a sun.nio.ch.Util$2)
    - locked <0x00000000a2fdb600> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kafka-producer-network-thread | prod-cisSegmenter-e0fcd3a4-d3a8-410f-b328-1a889e388605-StreamThread-1-0_9-producer" #59 daemon prio=5 os_prio=0 cpu=1444.17ms elapsed=291123.36s tid=0x00007f8a80014000 nid=0x59 runnable  [0x00007f8b482f9000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2fda490> (a sun.nio.ch.Util$2)
    - locked <0x00000000a2fda4a0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at org.apache.kafka.common.network.Selector.select(Selector.java:752)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:451)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:535)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:311)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"grpc-default-worker-ELG-3-1" #60 daemon prio=5 os_prio=0 cpu=1709.22ms elapsed=290547.40s tid=0x00007f8a6800a000 nid=0x5a runnable  [0x00007f8a5bdfc000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.1/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:120)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a2d2a368> (a io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
    - locked <0x00000000a2d2a268> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.1/SelectorImpl.java:136)
    at io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
    at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"pool-11-thread-1" #63 prio=5 os_prio=0 cpu=182.85ms elapsed=290547.22s tid=0x00007f8a4800d000 nid=0x5d runnable  [0x00007f8a5a6f8000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(java.base@11.0.1/Native Method)
    at java.net.SocketInputStream.socketRead(java.base@11.0.1/SocketInputStream.java:115)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:168)
    at java.net.SocketInputStream.read(java.base@11.0.1/SocketInputStream.java:140)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260)
    at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254)
    at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141)
    at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124)
    at java.util.concurrent.FutureTask.run(java.base@11.0.1/FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"kotlinx.coroutines.DefaultExecutor" #64 daemon prio=5 os_prio=0 cpu=6589.31ms elapsed=290547.07s tid=0x00007f8aa8016000 nid=0x5e waiting on condition  [0x00007f8a595f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a3205cc8> (a kotlinx.coroutines.DefaultExecutor)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:234)
    at kotlinx.coroutines.DefaultTimeSource.parkNanos(TimeSource.kt:31)
    at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:83)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Java2D Disposer" #69 daemon prio=10 os_prio=0 cpu=6.56ms elapsed=258769.35s tid=0x00007f8a80302000 nid=0x66 in Object.wait()  [0x00007f8a590f3000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:155)
    - waiting to re-lock in wait() <0x00000000a4c6d2a0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(java.base@11.0.1/ReferenceQueue.java:176)
    at sun.java2d.Disposer.run(java.desktop@11.0.1/Disposer.java:144)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-1" #71 daemon prio=5 os_prio=0 cpu=234.89ms elapsed=258767.46s tid=0x00007f8a80ef4800 nid=0x68 waiting on condition  [0x00007f8a46492000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-2" #77 daemon prio=5 os_prio=0 cpu=57.05ms elapsed=258405.07s tid=0x00007f8a80f7f000 nid=0x70 waiting on condition  [0x00007f8a5a9f9000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-3" #87 daemon prio=5 os_prio=0 cpu=36.56ms elapsed=258074.58s tid=0x00007f8a800d7800 nid=0x7d waiting on condition  [0x00007f8a45d8f000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-4" #89 daemon prio=5 os_prio=0 cpu=23.81ms elapsed=258074.02s tid=0x00007f8a80219800 nid=0x7f waiting on condition  [0x00007f8a45625000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-5" #94 daemon prio=5 os_prio=0 cpu=12.19ms elapsed=258071.29s tid=0x00007f8a8102b800 nid=0x89 waiting on condition  [0x00007f8a45423000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-6" #99 daemon prio=5 os_prio=0 cpu=16.35ms elapsed=257924.52s tid=0x00007f8a80fae000 nid=0x95 waiting on condition  [0x00007f8a5aafa000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-7" #106 daemon prio=5 os_prio=0 cpu=16.53ms elapsed=257854.52s tid=0x00007f8a8023f000 nid=0x9f waiting on condition  [0x00007f8a45524000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-8" #108 daemon prio=5 os_prio=0 cpu=12.27ms elapsed=257853.78s tid=0x00007f8a800ce000 nid=0xa1 waiting on condition  [0x00007f8a45021000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-9" #116 daemon prio=5 os_prio=0 cpu=15.76ms elapsed=257013.17s tid=0x00007f8a80d7f000 nid=0xb0 waiting on condition  [0x00007f8a5bffe000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"Tesseract-OCR-10" #119 daemon prio=5 os_prio=0 cpu=15.46ms elapsed=255999.37s tid=0x00007f8a803ab000 nid=0xb5 waiting on condition  [0x00007f8acc2d5000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    - parking to wait for  <0x00000000a1b01740> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.1/AbstractQueuedSynchronizer.java:2081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:1170)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.1/ThreadPoolExecutor.java:1054)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.1/ThreadPoolExecutor.java:1114)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.1/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.1/Thread.java:834)

"DefaultDispatcher-worker-4" #182 daemon prio=5 os_prio=0 cpu=5150.48ms elapsed=242667.86s tid=0x00007f8a6c103000 nid=0x11d waiting on condition  [0x00007f8acef9d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-2" #214 daemon prio=5 os_prio=0 cpu=4963.78ms elapsed=239781.03s tid=0x00007f8ab0001000 nid=0x145 waiting on condition  [0x00007f8a594f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-3" #226 daemon prio=5 os_prio=0 cpu=4947.72ms elapsed=239234.92s tid=0x00007f8aa4006800 nid=0x158 waiting on condition  [0x00007f8a591f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-4" #227 daemon prio=5 os_prio=0 cpu=14362.48ms elapsed=239234.72s tid=0x00007f8af400e000 nid=0x159 waiting on condition  [0x00007f8a44f20000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-1" #229 daemon prio=5 os_prio=0 cpu=4816.62ms elapsed=238528.77s tid=0x00007f8b2c00c800 nid=0x15b waiting on condition  [0x00007f8a46593000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-3" #230 daemon prio=5 os_prio=0 cpu=13946.05ms elapsed=238528.01s tid=0x00007f8ab0008800 nid=0x15c waiting on condition  [0x00007f8a4491e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-2" #236 daemon prio=5 os_prio=0 cpu=14150.12ms elapsed=238495.22s tid=0x00007f8b28004000 nid=0x163 waiting on condition  [0x00007f8a4481d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"DefaultDispatcher-worker-5" #237 daemon prio=5 os_prio=0 cpu=238247952.22ms elapsed=238285.87s tid=0x00007f8aa4004800 nid=0x164 runnable  [0x00007f8acf19e000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.1/EPollSelectorImpl.java:114)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.1/SelectorImpl.java:124)
    - locked <0x00000000a4e50c18> (a sun.nio.ch.Util$2)
    - locked <0x00000000a4e50ac8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.selectNow(java.base@11.0.1/SelectorImpl.java:146)
    at io.ktor.network.selector.ActorSelectorManager.process(ActorSelectorManager.kt:77)
    at io.ktor.network.selector.ActorSelectorManager$process$1.invokeSuspend(ActorSelectorManager.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742)

"DefaultDispatcher-worker-1" #238 daemon prio=5 os_prio=0 cpu=14330.80ms elapsed=238284.64s tid=0x00007f8b28005800 nid=0x165 waiting on condition  [0x00007f8a46290000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.1/LockSupport.java:357)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.doPark(CoroutineScheduler.kt:850)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.cpuWorkerIdle(CoroutineScheduler.kt:823)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:729)

"Attach Listener" #243 daemon prio=9 os_prio=0 cpu=0.39ms elapsed=2.99s tid=0x00007f8b20001000 nid=0x1c7 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=20416.12ms elapsed=291218.57s tid=0x00007f8b680da800 nid=0x16 runnable  

"GC Thread#0" os_prio=0 cpu=15765.00ms elapsed=291218.58s tid=0x00007f8b68029800 nid=0x11 runnable  

"GC Thread#1" os_prio=0 cpu=15710.68ms elapsed=291218.32s tid=0x00007f8b34001000 nid=0x20 runnable  

"GC Thread#2" os_prio=0 cpu=15687.51ms elapsed=291216.37s tid=0x00007f8b34024800 nid=0x30 runnable  

"GC Thread#3" os_prio=0 cpu=15598.91ms elapsed=291216.37s tid=0x00007f8b34025800 nid=0x31 runnable  

"G1 Main Marker" os_prio=0 cpu=128.24ms elapsed=291218.58s tid=0x00007f8b68050800 nid=0x12 runnable  

"G1 Conc#0" os_prio=0 cpu=44971.89ms elapsed=291218.58s tid=0x00007f8b68052800 nid=0x13 runnable  

"G1 Refine#0" os_prio=0 cpu=13.94ms elapsed=291218.58s tid=0x00007f8b680b2800 nid=0x14 runnable  

"G1 Refine#1" os_prio=0 cpu=3.17ms elapsed=291214.76s tid=0x00007f8b38001000 nid=0x40 runnable  

"G1 Refine#2" os_prio=0 cpu=0.14ms elapsed=291214.76s tid=0x00007f8a94001800 nid=0x41 runnable  

"G1 Refine#3" os_prio=0 cpu=0.10ms elapsed=291214.76s tid=0x00007f8aa0002000 nid=0x42 runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=46542.37ms elapsed=291218.58s tid=0x00007f8b680b4000 nid=0x15 runnable  
"VM Periodic Task Thread" os_prio=0 cpu=93364.16ms elapsed=291218.02s tid=0x00007f8b68cd1800 nid=0x24 waiting on condition  

JNI global refs: 90, weak refs: 3

And the output of the top 3 threads from top -n 1 -H -p <pid>:

top - 15:09:06 up 71 days, 10:43,  0 users,  load average: 1.27, 1.21, 1.23
Threads:  80 total,   1 running,  79 sleeping,   0 stopped,   0 zombie
%Cpu(s): 18.2 us, 18.2 sy,  0.0 ni, 63.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  32168.8 total,   8305.4 free,   4024.5 used,  19838.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  27675.9 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND    
  356 root      20   0 7541692 868496  33256 R  99.9   2.6   3991:28 DefaultDi+ 
    1 root      20   0 7541692 868496  33256 S   0.0   2.6   0:00.01 java       
   16 root      20   0 7541692 868496  33256 S   0.0   2.6   0:05.45 java       
   17 root      20   0 7541692 868496  33256 S   0.0   2.6   0:15.85 GC Thread+

I have a couple of different microservices that are failing in exactly the same way, with very different client-side code. This is the simpler code of the two cases:

  protected val client = HttpClient(CIO) {
    expectSuccess = false
  }

  protected fun URL.toMap() = runBlocking {
    client.call(this@toMap).use {
      val content = it.response.readText()
      if(!it.response.status.isSuccess()) {
        log.warn { "Non-successful HTTP response from URL ${this@toMap}: $content" }
      }

      objectMapper.readValue<Map<String, Any>>(content)
    }
  }

I don't have a reproducer for this -- it seems to happen randomly, but almost always happens eventually.

Possibly related GitHub issue, but that issue provides few thread details so hard to say whether its the same or not: https://github.com/ktorio/ktor/issues/1018.

rocketraman commented 5 years ago

@e5l I see the 1.2.0 release notes include "Fixed CIO client CPU utilization issue", but no reference to an issue id. Is that a fix for this issue?

e5l commented 5 years ago

Hi @rocketraman, yes the problem shouldn't happen, but the fix is still incomplete. We're expecting proper dispatcher fix from kotlinx.coroutines.

rocketraman commented 5 years ago

Thanks @e5l . Do you have a link to a big report in coroutines that I can track?

e5l commented 5 years ago

https://github.com/Kotlin/kotlinx.coroutines/issues/1046