search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.17k stars 3.57k forks source link

Pulsar 2.7.0+ KOP 2.7.2.x getPartitionedTopicMetadata timeout #11532

Closed xiaotongwang1 closed 1 year ago

xiaotongwang1 commented 3 years ago

web log

2021-08-02 14:50:20.473 [pulsar-web-47-4] INFO org.eclipse.jetty.server.RequestLog - 10.33.141.46 - - [02/Aug/2021:14:49:50 +0800] "GET /admin/v2/persistent/xxx/xxx/xsj.testtopic.0802.05-partition-13/stats?getPreciseBacklog=false&authoritative=false HTTP/1.1" 500 12267 "-" "Pulsar-Java-v2.7.0" 30260

2021-08-02 14:54:50.784 [pulsar-web-47-13] INFO org.eclipse.jetty.server.RequestLog - 10.33.141.44 - - [02/Aug/2021:14:54:21 +0800] "GET /admin/v2/persistent/WiseCloudDMQService/WiseCloudDMQAminService/xsj.testtopic.0729.01-partition-4/internalStats?metadata=false&authoritative=true HTTP/1.1" 200 4714 "-" "Pulsar-Java-v2.7.0" 29541 2021-08-02 14:54:50.784 [pulsar-web-47-32] INFO org.eclipse.jetty.server.RequestLog - 10.33.141.56 - - [02/Aug/2021:14:54:21 +0800] "GET /admin/v2/persistent/WiseCloudDMQService/WiseCloudDMQAminService/xsj.testtopic.0729.01/partitions HTTP/1.1" 200 17 "-" "Pulsar-Java-v2.7.0" 29542 2021-08-02 14:54:50.785 [pulsar-web-47-16] INFO org.eclipse.jetty.server.RequestLog - 10.33.141.56 - - [02/Aug/2021:14:54:21 +0800] "GET /admin/v2/persistent/WiseCloudDMQService/WiseCloudDMQAminService/xsj.testtopic.0729.01/partitions HTTP/1.1" 200 17 "-" "Pulsar-Java-v2.7.0" 29703

stack 1

"pulsar-web-47-4" #167 prio=5 os_prio=0 tid=0x00007fad5528d800 nid=0x2c6f waiting on condition [0x00007fad462c4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

stack 2

"pulsar-web-47-4" #167 prio=5 os_prio=0 tid=0x00007fad5528d800 nid=0x2c6f waiting on condition [0x00007fad462c4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

2021-08-02 15:05:51.825 [ForkJoinPool.commonPool-worker-27] WARN o.a.p.b.service.persistent.DispatchRateLimiter - Failed to get message-rate for persistent://WiseCloudDMQService/WiseCloudDMQAminService/xsj.testtopic.0730.05-partition-16 java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.broker.service.persistent.DispatchRateLimiter.getPolicies(DispatchRateLimiter.java:311) at org.apache.pulsar.broker.service.persistent.SubscribeRateLimiter.getPoliciesSubscribeRate(SubscribeRateLimiter.java:227) at org.apache.pulsar.broker.service.persistent.SubscribeRateLimiter.getPoliciesSubscribeRate(SubscribeRateLimiter.java:205) at org.apache.pulsar.broker.service.persistent.SubscribeRateLimiter.onSubscribeRateUpdate(SubscribeRateLimiter.java:167) at org.apache.pulsar.broker.service.persistent.SubscribeRateLimiter.onPoliciesUpdate(SubscribeRateLimiter.java:162) at org.apache.pulsar.broker.service.persistent.PersistentTopic.onPoliciesUpdate(PersistentTopic.java:2027) at org.apache.pulsar.broker.service.BrokerService.lambda$null$62(BrokerService.java:1602) at java.util.Optional.ifPresent(Optional.java:159) at org.apache.pulsar.broker.service.BrokerService.lambda$null$63(BrokerService.java:1602) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)

trace org.apache.pulsar.broker.service.persistent.DispatchRateLimiter getPolicies

---ts=2021-08-02 15:11:21;thread_name=ForkJoinPool.commonPool-worker-60;id=c47;is_daemon=true;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@18b4aac2 ---[30001.016609ms] org.apache.pulsar.broker.service.persistent.DispatchRateLimiter:getPolicies() +---[0.003785ms] org.apache.pulsar.common.naming.TopicName:get() #304 +---[0.002569ms] org.apache.pulsar.common.naming.TopicName:getNamespaceObject() #304 +---[0.002541ms] org.apache.pulsar.common.naming.NamespaceName:toString() #305 +---[0.003915ms] org.apache.pulsar.broker.web.PulsarWebResource:path() #305 +---[0.002655ms] org.apache.pulsar.broker.service.BrokerService:pulsar() #308 +---[0.002662ms] org.apache.pulsar.broker.PulsarService:getConfigurationCache() #308 +---[0.003075ms] org.apache.pulsar.broker.cache.ConfigurationCacheService:policiesCache() #310 +---[0.020181ms] org.apache.pulsar.zookeeper.ZooKeeperDataCache:getAsync() #310 +---[0.002184ms] org.apache.pulsar.broker.service.BrokerService:pulsar() #311 +---[0.002972ms] org.apache.pulsar.broker.PulsarService:getConfiguration() #311 `---[0.00194ms] org.apache.pulsar.broker.ServiceConfiguration:getZooKeeperOperationTimeoutSeconds() #311

BewareMyPower commented 3 years ago

Can you query the partitions number using pulsar-admin?

It looks like the request stuck at validateGlobalNamespaceOwnership, which eventually calls

pulsarService.getConfigurationCache().policiesCache().getAsync(path).thenAccept(policiesResult -> {

but the getAsync call timed out.

I think restarting the broker could solve the problem because cache will be refreshed, but what cause it is still to be figured out. Could you give some help? @codelipenghui

codelipenghui commented 3 years ago

@xiaotongwang1 Could you please provide the complete thread dump?

xiaotongwang1 commented 3 years ago

这个问题困扰了了几天，一添加权限，集群就挂死，被业务各种投诉，今天仔细看了代码，尝试修改后测试应该可以了，问题应该有两个，一个是pulsar自研的concurrent openhash map锁问题，后面尝试从2.8.0的代码拉取后，锁问题解决。但是集群挂死的问题依然存在，后面发现zookeeper data cache更新场景，单个topic的权限修改会导致namespace的策略更新，设计上感觉不太合理。cache更新操作的complete线程上，更新过程中存在循环调用cache get问题，导致线程池被占满，影响其他异步调用。

xiaotongwang1 commented 3 years ago

FULL STACK

2021-08-04 21:11:31 Full thread dump OpenJDK 64-Bit Server VM (25.272-b10 mixed mode):

"Attach Listener" #385 daemon prio=9 os_prio=0 tid=0x000055bb2ab1d000 nid=0x4a0 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"ForkJoinPool.commonPool-worker-25" #360 daemon prio=5 os_prio=0 tid=0x00007fc77c701000 nid=0x4108 waiting on condition [0x00007fc741e8d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-22" #358 daemon prio=5 os_prio=0 tid=0x00007fc7747a2800 nid=0x3d69 waiting on condition [0x00007fc741f8e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-21" #356 daemon prio=5 os_prio=0 tid=0x00007fc7824fe800 nid=0x3841 waiting on condition [0x00007fc74208f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-18" #353 daemon prio=5 os_prio=0 tid=0x000055bb2b3a6800 nid=0x2f78 waiting on condition [0x00007fc742190000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-8" #352 daemon prio=5 os_prio=0 tid=0x000055bb2b199000 nid=0x2f77 waiting on condition [0x00007fc742291000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-14" #349 daemon prio=5 os_prio=0 tid=0x000055bb2a482000 nid=0x2bfb waiting on condition [0x00007fc74319c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-7" #350 daemon prio=5 os_prio=0 tid=0x00007fc774426000 nid=0x2bfa waiting on condition [0x00007fc74309b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-15" #351 daemon prio=5 os_prio=0 tid=0x000055bb2a481000 nid=0x2bf9 waiting on condition [0x00007fc7600db000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-12" #348 daemon prio=5 os_prio=0 tid=0x000055bb2a04a800 nid=0x2bf8 waiting on condition [0x00007fc74329d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-4" #347 daemon prio=5 os_prio=0 tid=0x00007fc77c681000 nid=0x7e10 waiting on condition [0x00007fc778384000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-9" #345 daemon prio=5 os_prio=0 tid=0x000055bb2a480000 nid=0x7e0e waiting on condition [0x00007fc764d22000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-3" #344 daemon prio=5 os_prio=0 tid=0x000055bb2b3e1800 nid=0x7e0d waiting on condition [0x00007fc75ffda000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-8" #343 daemon prio=5 os_prio=0 tid=0x000055bb2bb77000 nid=0x5a5c waiting on condition [0x00007fc742393000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-6" #339 daemon prio=5 os_prio=0 tid=0x00007fc780cd6000 nid=0x57a2 waiting on condition [0x00007fc7602dd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-1" #341 daemon prio=5 os_prio=0 tid=0x00007fc774b9f000 nid=0x57a0 waiting on condition [0x00007fc74359e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-7" #338 daemon prio=5 os_prio=0 tid=0x000055bb2b379800 nid=0x518e waiting on condition [0x00007fc742694000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-6" #337 daemon prio=5 os_prio=0 tid=0x000055bb29c60800 nid=0x48c0 waiting on condition [0x00007fc742795000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-5" #336 daemon prio=5 os_prio=0 tid=0x000055bb29dab800 nid=0x3ff5 waiting on condition [0x00007fc742896000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-4" #335 daemon prio=5 os_prio=0 tid=0x000055bb2aaa2000 nid=0x372c waiting on condition [0x00007fc742997000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-2" #332 daemon prio=5 os_prio=0 tid=0x00007fc774c2d800 nid=0x2fa3 waiting on condition [0x00007fc742a97000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-11" #334 daemon prio=5 os_prio=0 tid=0x00007fc774ba1800 nid=0x2fa2 waiting on condition [0x00007fc768b4b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-0" #331 daemon prio=5 os_prio=0 tid=0x00007fc7808d6000 nid=0x2fa0 waiting on condition [0x00007fc7601dc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-3" #330 daemon prio=5 os_prio=0 tid=0x000055bb29f34800 nid=0x2e5e waiting on condition [0x00007fc768849000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"prometheus-stats-48-1" #329 prio=5 os_prio=0 tid=0x00007fc78410a800 nid=0x2596 waiting on condition [0x00007fc742b98000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-2" #328 daemon prio=5 os_prio=0 tid=0x000055bb29d88000 nid=0x2543 waiting on condition [0x00007fc742c9a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-5" #326 daemon prio=5 os_prio=0 tid=0x000055bb2a482800 nid=0x1f1b waiting on condition [0x00007fc742f9a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"bookkeeper-io-16-32" #322 prio=5 os_prio=0 tid=0x00007fc7747a0800 nid=0x1ef6 runnable [0x00007fc7586b5000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-31" #321 prio=5 os_prio=0 tid=0x00007fc775e34800 nid=0x1ef5 runnable [0x00007fc7587b6000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-30" #320 prio=5 os_prio=0 tid=0x00007fc7759a0800 nid=0x1ef4 runnable [0x00007fc7588b7000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-29" #319 prio=5 os_prio=0 tid=0x00007fc775e33800 nid=0x1ef3 runnable [0x00007fc7589b8000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-28" #318 prio=5 os_prio=0 tid=0x00007fc775e31800 nid=0x1ef2 runnable [0x00007fc758ab9000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-27" #317 prio=5 os_prio=0 tid=0x00007fc774877800 nid=0x1ef1 runnable [0x00007fc758bba000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-26" #316 prio=5 os_prio=0 tid=0x00007fc775bb9000 nid=0x1ef0 runnable [0x00007fc758cbb000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-25" #315 prio=5 os_prio=0 tid=0x00007fc775bb6800 nid=0x1eef runnable [0x00007fc758dbc000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-24" #314 prio=5 os_prio=0 tid=0x00007fc775bad000 nid=0x1eee runnable [0x00007fc758ebd000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-23" #313 prio=5 os_prio=0 tid=0x00007fc775bab000 nid=0x1eed runnable [0x00007fc758fbe000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-22" #312 prio=5 os_prio=0 tid=0x00007fc775baa000 nid=0x1eec runnable [0x00007fc7590bf000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-21" #311 prio=5 os_prio=0 tid=0x00007fc7741a6000 nid=0x1eeb runnable [0x00007fc7591c0000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-20" #310 prio=5 os_prio=0 tid=0x00007fc7741a4800 nid=0x1eea runnable [0x00007fc7592c1000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-19" #309 prio=5 os_prio=0 tid=0x00007fc77474b000 nid=0x1ee9 runnable [0x00007fc7593c2000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-18" #308 prio=5 os_prio=0 tid=0x00007fc77473d000 nid=0x1ee8 runnable [0x00007fc7594c3000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-17" #307 prio=5 os_prio=0 tid=0x00007fc775bff000 nid=0x1ee7 runnable [0x00007fc7595c4000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-16" #306 prio=5 os_prio=0 tid=0x00007fc775c01000 nid=0x1ee6 runnable [0x00007fc7596c5000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-15" #305 prio=5 os_prio=0 tid=0x00007fc775bff800 nid=0x1ee5 runnable [0x00007fc7597c6000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-14" #304 prio=5 os_prio=0 tid=0x00007fc775bfc800 nid=0x1ee4 runnable [0x00007fc7598c7000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-13" #303 prio=5 os_prio=0 tid=0x00007fc775bf8000 nid=0x1ee3 runnable [0x00007fc7599c8000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-12" #302 prio=5 os_prio=0 tid=0x00007fc775bf6000 nid=0x1ee2 runnable [0x00007fc759ac9000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-11" #301 prio=5 os_prio=0 tid=0x00007fc775bf4000 nid=0x1ee1 runnable [0x00007fc759bca000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-10" #300 prio=5 os_prio=0 tid=0x00007fc775bf2000 nid=0x1ee0 runnable [0x00007fc759ccb000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-9" #299 prio=5 os_prio=0 tid=0x00007fc7759be800 nid=0x1edf runnable [0x00007fc759dcc000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-8" #298 prio=5 os_prio=0 tid=0x00007fc7759bc800 nid=0x1ede runnable [0x00007fc759ecd000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-7" #297 prio=5 os_prio=0 tid=0x00007fc7759ba800 nid=0x1edd runnable [0x00007fc759fce000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-6" #296 prio=5 os_prio=0 tid=0x00007fc7759b8800 nid=0x1edc runnable [0x00007fc75a0cf000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-5" #295 prio=5 os_prio=0 tid=0x00007fc7759b7000 nid=0x1edb runnable [0x00007fc75a1d0000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-4" #294 prio=5 os_prio=0 tid=0x00007fc7759b5000 nid=0x1eda runnable [0x00007fc75a2d1000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-3" #293 prio=5 os_prio=0 tid=0x00007fc7759b3800 nid=0x1ed9 runnable [0x00007fc75a3d2000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-2" #292 prio=5 os_prio=0 tid=0x00007fc774210800 nid=0x1ed8 runnable [0x00007fc75a4d3000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"bookkeeper-io-16-1" #291 prio=5 os_prio=0 tid=0x000055bb2b693800 nid=0x1ed7 runnable [0x00007fc75a5d4000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"ForkJoinPool.commonPool-worker-10" #290 daemon prio=5 os_prio=0 tid=0x000055bb2bb27000 nid=0x1df4 waiting on condition [0x00007fc75fed9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ForkJoinPool.commonPool-worker-13" #285 daemon prio=5 os_prio=0 tid=0x00007fc774545800 nid=0x1def waiting on condition [0x00007fc7603de000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-32" #284 prio=5 os_prio=0 tid=0x00007fc7744b1800 nid=0x1dee waiting on condition [0x00007fc7604df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-31" #283 prio=5 os_prio=0 tid=0x00007fc7744af000 nid=0x1ded waiting on condition [0x00007fc7605e0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-30" #282 prio=5 os_prio=0 tid=0x000055bb2baf5800 nid=0x1dec waiting on condition [0x00007fc7606e1000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-29" #281 prio=5 os_prio=0 tid=0x000055bb2bb31800 nid=0x1deb waiting on condition [0x00007fc7607e2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-28" #280 prio=5 os_prio=0 tid=0x000055bb2bb2f000 nid=0x1dea waiting on condition [0x00007fc7608e3000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-27" #279 prio=5 os_prio=0 tid=0x000055bb2bb2d000 nid=0x1de9 waiting for monitor entry [0x00007fc7609e5000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"pulsar-io-26-26" #278 prio=5 os_prio=0 tid=0x00007fc775743800 nid=0x1de7 waiting on condition [0x00007fc760ae5000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-25" #277 prio=5 os_prio=0 tid=0x00007fc775c58000 nid=0x1de6 waiting for monitor entry [0x00007fc760be7000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"ExpirationReaper-group-coordinator-delayed-heartbeat" #276 prio=5 os_prio=0 tid=0x00007fc77c222000 nid=0x1de5 waiting on condition [0x00007fc760ce8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ExpirationReaper-group-coordinator-delayed-join" #275 prio=5 os_prio=0 tid=0x00007fc77cc2c800 nid=0x1de4 waiting on condition [0x00007fc760de9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-24" #274 prio=5 os_prio=0 tid=0x000055bb2c2f1800 nid=0x1de3 waiting for monitor entry [0x00007fc7610ea000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"pulsar-io-26-23" #273 prio=5 os_prio=0 tid=0x000055bb2c2ef800 nid=0x1de2 waiting for monitor entry [0x00007fc7611eb000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"pulsar-io-26-22" #272 prio=5 os_prio=0 tid=0x000055bb2c2ec800 nid=0x1de1 waiting on condition [0x00007fc7612eb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-21" #271 prio=5 os_prio=0 tid=0x00007fc780800800 nid=0x1de0 waiting on condition [0x00007fc7613ec000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-19" #270 prio=5 os_prio=0 tid=0x000055bb2c2e9800 nid=0x1ddf waiting on condition [0x00007fc7614ed000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-20" #269 prio=5 os_prio=0 tid=0x00007fc781017000 nid=0x1dde waiting on condition [0x00007fc7615ee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-15-0" #268 prio=5 os_prio=0 tid=0x000055bb2bb42000 nid=0x1ddd waiting on condition [0x00007fc7616f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-14-0" #267 prio=5 os_prio=0 tid=0x000055bb2bb40000 nid=0x1ddc waiting on condition [0x00007fc7617f1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-13-0" #266 prio=5 os_prio=0 tid=0x000055bb2bb3d800 nid=0x1ddb waiting on condition [0x00007fc7618f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-12-0" #265 prio=5 os_prio=0 tid=0x000055bb2bb3b800 nid=0x1dda waiting on condition [0x00007fc7619f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-11-0" #264 prio=5 os_prio=0 tid=0x000055bb2bb39000 nid=0x1dd9 waiting on condition [0x00007fc761af4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-10-0" #263 prio=5 os_prio=0 tid=0x000055bb2bb36800 nid=0x1dd8 waiting on condition [0x00007fc761bf5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-9-0" #262 prio=5 os_prio=0 tid=0x000055bb2bb34800 nid=0x1dd7 waiting on condition [0x00007fc761cf6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-8-0" #261 prio=5 os_prio=0 tid=0x000055bb2ba6e800 nid=0x1dd6 waiting on condition [0x00007fc761df7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-7-0" #260 prio=5 os_prio=0 tid=0x000055bb2ba6c000 nid=0x1dd5 waiting on condition [0x00007fc761ef8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-6-0" #259 prio=5 os_prio=0 tid=0x000055bb2ba69800 nid=0x1dd4 waiting on condition [0x00007fc761ff9000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-5-0" #258 prio=5 os_prio=0 tid=0x000055bb2ba67800 nid=0x1dd3 waiting on condition [0x00007fc7620fa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-4-0" #257 prio=5 os_prio=0 tid=0x000055bb2ba65000 nid=0x1dd2 waiting on condition [0x00007fc7621fb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"group-coordinator-executor-OrderedScheduler-3-0" #256 prio=5 os_prio=0 tid=0x000055bb2ba63000 nid=0x1dd1 waiting on condition [0x00007fc7622fc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-2-0" #255 prio=5 os_prio=0 tid=0x000055bb2ba60800 nid=0x1dd0 waiting on condition [0x00007fc7623fd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-1-0" #254 prio=5 os_prio=0 tid=0x000055bb2c2fd000 nid=0x1dcf waiting on condition [0x00007fc7624fe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"group-coordinator-executor-OrderedScheduler-0-0" #253 prio=5 os_prio=0 tid=0x000055bb2c2fb800 nid=0x1dce waiting on condition [0x00007fc7625ff000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-client-io-55-1" #251 prio=5 os_prio=0 tid=0x000055bb2bb01000 nid=0x1dcd runnable [0x00007fc762700000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:132) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"pulsar-io-26-18" #250 prio=5 os_prio=0 tid=0x000055bb2bafe000 nid=0x1dcc waiting on condition [0x00007fc762800000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-17" #249 prio=5 os_prio=0 tid=0x000055bb2baa8000 nid=0x1dcb waiting on condition [0x00007fc762901000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-16" #248 prio=5 os_prio=0 tid=0x000055bb2baa6000 nid=0x1dca waiting for monitor entry [0x00007fc762a02000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)

"pulsar-io-26-15" #247 prio=5 os_prio=0 tid=0x00007fc774c76000 nid=0x1dc9 waiting on condition [0x00007fc762b03000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-13" #246 prio=5 os_prio=0 tid=0x000055bb2bad7000 nid=0x1dc8 waiting on condition [0x00007fc762c04000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-14" #245 prio=5 os_prio=0 tid=0x00007fc774c74800 nid=0x1dc7 waiting on condition [0x00007fc762d05000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-12" #244 prio=5 os_prio=0 tid=0x00007fc7759de800 nid=0x1dc6 waiting on condition [0x00007fc763009000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-11" #243 prio=5 os_prio=0 tid=0x00007fc7a772e000 nid=0x1dc5 waiting for monitor entry [0x00007fc76310b000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"pulsar-io-26-10" #242 prio=5 os_prio=0 tid=0x00007fc7759dd800 nid=0x1dc4 waiting on condition [0x00007fc76320b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-9" #241 prio=5 os_prio=0 tid=0x00007fc7851ea000 nid=0x1dc3 waiting on condition [0x00007fc76330c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-metadata-store-callback-23-1" #240 prio=5 os_prio=0 tid=0x00007fc780e81000 nid=0x1dc2 waiting on condition [0x00007fc76360e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-8" #239 prio=5 os_prio=0 tid=0x000055bb2ba94000 nid=0x1dc1 waiting on condition [0x00007fc76370e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-6" #237 prio=5 os_prio=0 tid=0x000055bb2ba92000 nid=0x1dc0 waiting on condition [0x00007fc76380f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-7" #238 prio=5 os_prio=0 tid=0x000055bb2bab1000 nid=0x1dbf waiting on condition [0x00007fc763910000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-5" #236 prio=5 os_prio=0 tid=0x000055bb2baaf800 nid=0x1dbe waiting for monitor entry [0x00007fc763a12000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.concurrent.ConcurrentHashMap.clear(ConcurrentHashMap.java:1193)

"pulsar-io-26-4" #235 prio=5 os_prio=0 tid=0x000055bb2ba76000 nid=0x1dbd waiting on condition [0x00007fc763b12000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-3" #234 prio=5 os_prio=0 tid=0x00007fc774541800 nid=0x1dbc waiting on condition [0x00007fc763c13000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-2" #232 prio=5 os_prio=0 tid=0x000055bb2ba58800 nid=0x1dbb waiting on condition [0x00007fc763d14000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-io-26-1" #233 prio=5 os_prio=0 tid=0x00007fc775359000 nid=0x1dba waiting on condition [0x00007fc763e15000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-2-16" #231 prio=5 os_prio=0 tid=0x00007fc775043800 nid=0x1db9 waiting for monitor entry [0x00007fc764117000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-15" #230 prio=5 os_prio=0 tid=0x00007fc77466d000 nid=0x1db8 waiting on condition [0x00007fc764218000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-2-12" #229 prio=5 os_prio=0 tid=0x000055bb2b653000 nid=0x1db7 waiting for monitor entry [0x00007fc764319000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-13" #228 prio=5 os_prio=0 tid=0x000055bb2b635800 nid=0x1db6 waiting for monitor entry [0x00007fc76441a000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-14" #227 prio=5 os_prio=0 tid=0x000055bb2b62a000 nid=0x1db5 waiting for monitor entry [0x00007fc76451b000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-11" #226 prio=5 os_prio=0 tid=0x00007fc782115800 nid=0x1db4 waiting for monitor entry [0x00007fc76461c000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-10" #225 prio=5 os_prio=0 tid=0x000055bb2b627800 nid=0x1db3 waiting on condition [0x00007fc76471d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-2-9" #224 prio=5 os_prio=0 tid=0x000055bb2b623000 nid=0x1db2 waiting for monitor entry [0x00007fc76481e000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-8" #223 prio=5 os_prio=0 tid=0x000055bb2b61c800 nid=0x1db0 waiting for monitor entry [0x00007fc76491f000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-7" #222 prio=5 os_prio=0 tid=0x000055bb2b619800 nid=0x1daf waiting on condition [0x00007fc764a20000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-2-6" #221 prio=5 os_prio=0 tid=0x000055bb2ae62800 nid=0x1dae waiting for monitor entry [0x00007fc764b21000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-5" #220 prio=5 os_prio=0 tid=0x00007fc7760fb800 nid=0x1dad waiting on condition [0x00007fc764c22000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-32" #218 prio=5 os_prio=0 tid=0x000055bb2b7d4800 nid=0x1dab waiting on condition [0x00007fc764e22000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-31" #217 prio=5 os_prio=0 tid=0x000055bb2b7d0000 nid=0x1daa waiting on condition [0x00007fc764f24000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"pulsar-web-47-30" #216 prio=5 os_prio=0 tid=0x000055bb2b7cb000 nid=0x1da9 waiting on condition [0x00007fc765024000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-29" #215 prio=5 os_prio=0 tid=0x000055bb2b7c8000 nid=0x1da8 waiting on condition [0x00007fc765126000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-28" #214 prio=5 os_prio=0 tid=0x000055bb2b7c9000 nid=0x1da7 waiting on condition [0x00007fc765227000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-27" #213 prio=5 os_prio=0 tid=0x000055bb2b7c5800 nid=0x1da6 waiting on condition [0x00007fc765328000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-26" #212 prio=5 os_prio=0 tid=0x000055bb2b7c2000 nid=0x1da5 runnable [0x00007fc76542a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pulsar-web-47-25" #211 prio=5 os_prio=0 tid=0x000055bb2b7bf800 nid=0x1da4 waiting on condition [0x00007fc765529000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-24" #210 prio=5 os_prio=0 tid=0x000055bb2b7bd800 nid=0x1da3 waiting on condition [0x00007fc76562b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-23" #209 prio=5 os_prio=0 tid=0x000055bb2b7bb000 nid=0x1da2 waiting on condition [0x00007fc76572b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-22" #208 prio=5 os_prio=0 tid=0x000055bb2b7b9000 nid=0x1da1 waiting on condition [0x00007fc76582d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-21" #207 prio=5 os_prio=0 tid=0x000055bb2ba49800 nid=0x1da0 waiting on condition [0x00007fc76592d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-20" #206 prio=5 os_prio=0 tid=0x000055bb2ba47800 nid=0x1d9f waiting on condition [0x00007fc765a2f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-19" #205 prio=5 os_prio=0 tid=0x000055bb2ba45000 nid=0x1d9e waiting on condition [0x00007fc765b30000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-18" #204 prio=5 os_prio=0 tid=0x000055bb2b7e8000 nid=0x1d9d waiting on condition [0x00007fc765c31000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-7" #203 prio=5 os_prio=0 tid=0x000055bb2b7e5800 nid=0x1d9c waiting on condition [0x00007fc765d33000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-6" #202 prio=5 os_prio=0 tid=0x000055bb2b667000 nid=0x1d9b waiting on condition [0x00007fc765e34000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-5" #201 prio=5 os_prio=0 tid=0x000055bb2b664800 nid=0x1d9a waiting on condition [0x00007fc765f35000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-4" #200 prio=5 os_prio=0 tid=0x000055bb2b7b3800 nid=0x1d99 waiting on condition [0x00007fc766036000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-3" #199 prio=5 os_prio=0 tid=0x000055bb2b7b1800 nid=0x1d98 waiting on condition [0x00007fc766137000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-2" #198 prio=5 os_prio=0 tid=0x000055bb2b66a000 nid=0x1d97 waiting on condition [0x00007fc766238000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"jersey-client-async-executor-1" #197 prio=5 os_prio=0 tid=0x000055bb2b669000 nid=0x1d96 waiting on condition [0x00007fc766339000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-17" #196 prio=5 os_prio=0 tid=0x000055bb2b78e800 nid=0x1d95 waiting on condition [0x00007fc766438000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"pulsar-web-47-16" #195 prio=5 os_prio=0 tid=0x000055bb2b74b000 nid=0x1d94 waiting on condition [0x00007fc76753a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"delayer-52-1" #194 prio=5 os_prio=0 tid=0x00007fc780acf000 nid=0x1d93 waiting on condition [0x00007fc76763d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"AsyncHttpClient-54-1" #193 prio=5 os_prio=0 tid=0x00007fc7743d1000 nid=0x1d92 runnable [0x00007fc76773e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"jersey-client-async-executor-0" #192 prio=5 os_prio=0 tid=0x00007fc78500e000 nid=0x1d91 waiting on condition [0x00007fc767a3f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-15" #191 prio=5 os_prio=0 tid=0x000055bb2b2e7800 nid=0x1d74 waiting on condition [0x00007fc767b3e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-14" #190 prio=5 os_prio=0 tid=0x000055bb2b2e7000 nid=0x1d73 waiting on condition [0x00007fc767c40000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"AsyncHttpClient-timer-53-1" #189 prio=5 os_prio=0 tid=0x00007fc77487f000 nid=0x1d72 waiting on condition [0x00007fc767f42000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"Connector-Scheduler-58cec85b-1" #188 prio=5 os_prio=0 tid=0x000055bb2b338000 nid=0x1d63 waiting on condition [0x00007fc768043000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-13" #187 prio=5 os_prio=0 tid=0x000055bb2b32b800 nid=0x1d62 waiting on condition [0x00007fc768142000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-modular-load-manager-37-1" #186 prio=5 os_prio=0 tid=0x00007fc775208800 nid=0x1d61 waiting on condition [0x00007fc768245000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ExpirationReaper-topic" #185 prio=5 os_prio=0 tid=0x000055bb2b320800 nid=0x1d60 waiting on condition [0x00007fc768346000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"AsyncHttpClient-timer-50-1" #184 prio=5 os_prio=0 tid=0x000055bb2b2af800 nid=0x1d59 waiting on condition [0x00007fc768447000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"pulsar-2-4" #183 prio=5 os_prio=0 tid=0x00007fc782d9f000 nid=0x1d35 waiting for monitor entry [0x00007fc768748000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-3" #181 prio=5 os_prio=0 tid=0x00007fc775ba8800 nid=0x1cef waiting for monitor entry [0x00007fc76894a000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-2-2" #180 prio=5 os_prio=0 tid=0x00007fc775ba7800 nid=0x1ce6 waiting for monitor entry [0x00007fc768a4b000] java.lang.Thread.State: BLOCKED (on object monitor) at io.streamnative.pulsar.handlers.kop.KafkaTopicManager.registerProducerInPersistentTopic(KafkaTopicManager.java:312)

"pulsar-web-47-12-acceptor-0@6cd2cb2-PulsarServerConnector@68ba310d{SSL, (ssl, http/1.1)}{0.0.0.0:8443}" #178 prio=3 os_prio=0 tid=0x00007fc7763a0000 nid=0x1cb9 runnable [0x00007fc768e4d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)

"pulsar-web-47-11" #177 prio=5 os_prio=0 tid=0x00007fc774bb5800 nid=0x1cb8 runnable [0x00007fc768f4e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

"pulsar-web-47-10-acceptor-0@182cc69e-PulsarServerConnector@58cec85b{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}" #176 prio=3 os_prio=0 tid=0x000055bb2b395000 nid=0x1cb7 runnable [0x00007fc76904f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)

"pulsar-web-47-9" #175 prio=5 os_prio=0 tid=0x000055bb2b10e000 nid=0x1cb6 waiting on condition [0x00007fc76914f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Session-HouseKeeper-6df3e44c-1" #174 prio=5 os_prio=0 tid=0x00007fc7a6819800 nid=0x1c85 waiting on condition [0x00007fc769a51000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-8" #173 prio=5 os_prio=0 tid=0x00007fc784e1e800 nid=0x1c83 waiting on condition [0x00007fc769b50000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-7" #172 prio=5 os_prio=0 tid=0x00007fc784e1d000 nid=0x1c82 waiting on condition [0x00007fc769c51000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-6" #171 prio=5 os_prio=0 tid=0x00007fc784e1b000 nid=0x1c81 waiting on condition [0x00007fc769d53000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-5" #170 prio=5 os_prio=0 tid=0x00007fc784e19000 nid=0x1c80 waiting on condition [0x00007fc769e54000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-4" #169 prio=5 os_prio=0 tid=0x00007fc784e17000 nid=0x1c7f waiting on condition [0x00007fc769f55000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-3" #168 prio=5 os_prio=0 tid=0x00007fc784e15800 nid=0x1c7e waiting on condition [0x00007fc76a056000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-web-47-2" #167 prio=5 os_prio=0 tid=0x00007fc784e14800 nid=0x1c7d waiting on condition [0x00007fc76a156000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"pulsar-web-47-1" #166 prio=5 os_prio=0 tid=0x00007fc784f14000 nid=0x1c7c waiting on condition [0x00007fc76b92f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-backlog-quota-checker-34-1" #165 prio=5 os_prio=0 tid=0x00007fc7807f4000 nid=0x1c5f waiting on condition [0x00007fc76a459000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"ledger-full-monitor-33-1" #164 prio=5 os_prio=0 tid=0x00007fc7801fc800 nid=0x1c5e waiting on condition [0x00007fc76a55a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"consumed-Ledgers-monitor-32-1" #163 prio=5 os_prio=0 tid=0x00007fc780233000 nid=0x1c5d waiting on condition [0x00007fc76a65b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-publish-buffer-monitor-31-1" #162 prio=5 os_prio=0 tid=0x00007fc7806e8000 nid=0x1c5c waiting on condition [0x00007fc76a75c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-compaction-monitor-30-1" #161 prio=5 os_prio=0 tid=0x00007fc780541800 nid=0x1c5b waiting on condition [0x00007fc76e7b9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-msg-expiry-monitor-29-1" #160 prio=5 os_prio=0 tid=0x00007fc7805c9000 nid=0x1c5a waiting on condition [0x00007fc76e8ba000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-inactivity-monitor-28-1" #159 prio=5 os_prio=0 tid=0x00007fc7802b3000 nid=0x1c59 waiting on condition [0x00007fc76e9bb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-stats-updater-27-1" #158 prio=5 os_prio=0 tid=0x00007fc774bb9800 nid=0x1c58 waiting on condition [0x00007fc76eabc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-acceptor-25-1" #157 prio=5 os_prio=0 tid=0x00007fc774bb8800 nid=0x1c57 runnable [0x00007fc76ebbd000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"pulsar-2-1" #156 prio=5 os_prio=0 tid=0x00007fc774c58000 nid=0x1c56 waiting on condition [0x00007fc76ecbe000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"epollEventLoopGroup-42-1" #154 prio=10 os_prio=0 tid=0x000055bb2ad7f800 nid=0x1c55 runnable [0x00007fc76edbf000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"main-EventThread" #153 daemon prio=5 os_prio=0 tid=0x000055bb2ad7d000 nid=0x1c54 waiting on condition [0x00007fc76bc32000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"main-SendThread(10.33.141.44:2281)" #152 daemon prio=5 os_prio=0 tid=0x000055bb2ad7b800 nid=0x1c53 waiting on condition [0x00007fc76e575000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-15-0" #151 prio=5 os_prio=0 tid=0x00007fc7a5439800 nid=0x1c52 waiting on condition [0x00007fc76cd43000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-14-0" #150 prio=5 os_prio=0 tid=0x00007fc7a52d6800 nid=0x1c51 waiting on condition [0x00007fc76ca40000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-13-0" #149 prio=5 os_prio=0 tid=0x00007fc7a673f000 nid=0x1c50 waiting on condition [0x00007fc76cb41000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-12-0" #148 prio=5 os_prio=0 tid=0x00007fc7a674b800 nid=0x1c4f waiting on condition [0x00007fc76c73d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-11-0" #147 prio=5 os_prio=0 tid=0x00007fc7a6748000 nid=0x1c4e waiting on condition [0x00007fc76c83e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-10-0" #146 prio=5 os_prio=0 tid=0x00007fc7a6742800 nid=0x1c4d waiting on condition [0x00007fc76c137000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-9-0" #145 prio=5 os_prio=0 tid=0x00007fc7a674a000 nid=0x1c4c waiting on condition [0x00007fc76c63c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-8-0" #144 prio=5 os_prio=0 tid=0x00007fc7a6740800 nid=0x1c4b waiting on condition [0x00007fc76bd33000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-7-0" #143 prio=5 os_prio=0 tid=0x00007fc7a6737800 nid=0x1c4a waiting on condition [0x00007fc76cc42000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-6-0" #142 prio=5 os_prio=0 tid=0x00007fc7a6746000 nid=0x1c49 waiting on condition [0x00007fc76c93f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-5-0" #141 prio=5 os_prio=0 tid=0x00007fc7a674d800 nid=0x1c48 waiting on condition [0x00007fc76c43a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-4-0" #140 prio=5 os_prio=0 tid=0x00007fc7a6739800 nid=0x1c47 waiting on condition [0x00007fc76bf35000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-3-0" #139 prio=5 os_prio=0 tid=0x00007fc7a673d000 nid=0x1c46 waiting on condition [0x00007fc76c036000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-2-0" #138 prio=5 os_prio=0 tid=0x00007fc7a673b000 nid=0x1c45 waiting on condition [0x00007fc76c238000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-1-0" #137 prio=5 os_prio=0 tid=0x00007fc7a5438800 nid=0x1c44 waiting on condition [0x00007fc76c339000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-0-0" #136 prio=5 os_prio=0 tid=0x00007fc7a6744800 nid=0x1c43 waiting on condition [0x00007fc76c53b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientScheduler-OrderedScheduler-0-0" #135 prio=5 os_prio=0 tid=0x00007fc7a670c000 nid=0x1c42 waiting on condition [0x00007fc76be34000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-load-manager-1-1" #116 prio=5 os_prio=0 tid=0x00007fc78487c800 nid=0x1c29 waiting on condition [0x00007fc76e676000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Curator-Framework-0" #114 daemon prio=5 os_prio=0 tid=0x000055bb2a9e9000 nid=0x1c23 waiting on condition [0x00007fc76e045000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"epollEventLoopGroup-35-1" #113 prio=10 os_prio=0 tid=0x00007fc7742d8000 nid=0x1c22 runnable [0x00007fc76e146000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"main-EventThread" #112 daemon prio=5 os_prio=0 tid=0x000055bb2a9e4000 nid=0x1c21 waiting on condition [0x00007fc76e247000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"main-SendThread(10.33.141.44:2281)" #111 daemon prio=5 os_prio=0 tid=0x000055bb2a9e3000 nid=0x1c20 waiting on condition [0x00007fc76e348000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Curator-ConnectionStateManager-0" #110 daemon prio=5 os_prio=0 tid=0x000055bb2a847000 nid=0x1c1f waiting on condition [0x00007fc76e449000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"scram-user-reload" #109 prio=5 os_prio=0 tid=0x000055bb2a31a000 nid=0x1c1e in Object.wait() [0x00007fc76eec0000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552)

"broker-topic-workers-OrderedScheduler-7-0" #108 prio=5 os_prio=0 tid=0x00007fc7a52c1000 nid=0x1c1d waiting on condition [0x00007fc76efc1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-6-0" #107 prio=5 os_prio=0 tid=0x00007fc7a52bf000 nid=0x1c1c waiting on condition [0x00007fc76f0c2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-5-0" #106 prio=5 os_prio=0 tid=0x00007fc7a52bd000 nid=0x1c1b waiting on condition [0x00007fc76f1c3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-4-0" #105 prio=5 os_prio=0 tid=0x00007fc7a52bb800 nid=0x1c1a waiting on condition [0x00007fc76f2c4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-3-0" #104 prio=5 os_prio=0 tid=0x00007fc7a52b9800 nid=0x1c19 waiting on condition [0x00007fc76f3c5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-2-0" #103 prio=5 os_prio=0 tid=0x00007fc7a52b8000 nid=0x1c18 waiting on condition [0x00007fc76f4c6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-1-0" #102 prio=5 os_prio=0 tid=0x00007fc7a626e800 nid=0x1c16 waiting on condition [0x00007fc76f5c7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"broker-topic-workers-OrderedScheduler-0-0" #101 prio=5 os_prio=0 tid=0x00007fc7a5e34000 nid=0x1c15 waiting on condition [0x00007fc76f6c8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-cache-eviction-22-1" #100 prio=5 os_prio=0 tid=0x00007fc7a61b1800 nid=0x1c14 runnable [0x00007fc76f7c9000] java.lang.Thread.State: RUNNABLE at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.getEarlierReadPositionForActiveCursors(ManagedLedgerImpl.java:1857) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.doCacheEviction(ManagedLedgerImpl.java:1846) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$doCacheEviction$5(ManagedLedgerFactoryImpl.java:289) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$$Lambda$87/75928167.accept(Unknown Source) at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.doCacheEviction(ManagedLedgerFactoryImpl.java:285)

"bookkeeper-ml-workers-OrderedExecutor-7-0" #99 prio=5 os_prio=0 tid=0x00007fc7a61a6000 nid=0x1c13 waiting on condition [0x00007fc76f8ca000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-6-0" #98 prio=5 os_prio=0 tid=0x00007fc7a61a4000 nid=0x1c07 waiting on condition [0x00007fc76f9cb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-5-0" #97 prio=5 os_prio=0 tid=0x00007fc7a61a2000 nid=0x1c06 waiting on condition [0x00007fc76facc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-4-0" #96 prio=5 os_prio=0 tid=0x00007fc7a61a0000 nid=0x1c05 waiting on condition [0x00007fc76fbcc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-3-0" #95 prio=5 os_prio=0 tid=0x00007fc7a619e000 nid=0x1c04 waiting on condition [0x00007fc76fcce000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-2-0" #94 prio=5 os_prio=0 tid=0x00007fc7a619c800 nid=0x1c03 waiting on condition [0x00007fc76fdcf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-1-0" #93 prio=5 os_prio=0 tid=0x00007fc7a619a800 nid=0x1c02 waiting on condition [0x00007fc76fed0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-workers-OrderedExecutor-0-0" #92 prio=5 os_prio=0 tid=0x00007fc7a6198800 nid=0x1c01 waiting on condition [0x00007fc76ffd1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-7-0" #91 prio=5 os_prio=0 tid=0x00007fc7a6186000 nid=0x1c00 waiting on condition [0x00007fc7700d2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-6-0" #90 prio=5 os_prio=0 tid=0x00007fc7a6164000 nid=0x1bff waiting on condition [0x00007fc7701d3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-5-0" #89 prio=5 os_prio=0 tid=0x00007fc7a6162000 nid=0x1bfe waiting on condition [0x00007fc7702d4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-4-0" #88 prio=5 os_prio=0 tid=0x00007fc7a6160000 nid=0x1bfd waiting on condition [0x00007fc7703d5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-3-0" #87 prio=5 os_prio=0 tid=0x00007fc7a615e000 nid=0x1bfc waiting on condition [0x00007fc7704d6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-2-0" #86 prio=5 os_prio=0 tid=0x00007fc7a615c800 nid=0x1bfb waiting on condition [0x00007fc7705d7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-1-0" #85 prio=5 os_prio=0 tid=0x00007fc7a6129800 nid=0x1bfa waiting on condition [0x00007fc7706d8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"bookkeeper-ml-scheduler-OrderedScheduler-0-0" #84 prio=5 os_prio=0 tid=0x00007fc7a6128800 nid=0x1bf9 waiting on condition [0x00007fc7707d9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"epollEventLoopGroup-15-1" #82 prio=10 os_prio=0 tid=0x000055bb2a241000 nid=0x1b55 runnable [0x00007fc771adb000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"main-EventThread" #81 daemon prio=5 os_prio=0 tid=0x00007fc7844ca000 nid=0x1b51 waiting on condition [0x00007fc771bdc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"main-SendThread(10.33.141.46:2281)" #80 daemon prio=5 os_prio=0 tid=0x00007fc7844c9000 nid=0x1b50 waiting on condition [0x00007fc771cdd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

xiaotongwang1 commented 3 years ago

"BookKeeperClientWorker-OrderedExecutor-15-0" #79 prio=5 os_prio=0 tid=0x00007fc784791000 nid=0x1b35 waiting on condition [0x00007fc771fde000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-14-0" #78 prio=5 os_prio=0 tid=0x00007fc78478f800 nid=0x1b34 waiting on condition [0x00007fc7720df000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-13-0" #77 prio=5 os_prio=0 tid=0x00007fc78478d800 nid=0x1b33 waiting on condition [0x00007fc7721e0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-12-0" #76 prio=5 os_prio=0 tid=0x00007fc78478b000 nid=0x1b32 waiting on condition [0x00007fc7722e1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-11-0" #75 prio=5 os_prio=0 tid=0x00007fc7842ea800 nid=0x1b30 waiting on condition [0x00007fc7723e2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-10-0" #74 prio=5 os_prio=0 tid=0x00007fc7842e8000 nid=0x1b2e waiting on condition [0x00007fc7724e3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-9-0" #73 prio=5 os_prio=0 tid=0x00007fc7842e6800 nid=0x1b2c waiting on condition [0x00007fc7725e4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-8-0" #72 prio=5 os_prio=0 tid=0x00007fc7842e4800 nid=0x1b2a waiting on condition [0x00007fc7726e5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-7-0" #71 prio=5 os_prio=0 tid=0x00007fc7842e3800 nid=0x1b29 waiting on condition [0x00007fc7727e6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-6-0" #70 prio=5 os_prio=0 tid=0x00007fc7842b8800 nid=0x1b28 waiting on condition [0x00007fc7728e7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-5-0" #69 prio=5 os_prio=0 tid=0x00007fc7842b6800 nid=0x1b25 waiting on condition [0x00007fc7729e8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-4-0" #68 prio=5 os_prio=0 tid=0x00007fc7842b5000 nid=0x1b23 waiting on condition [0x00007fc772ae9000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-3-0" #67 prio=5 os_prio=0 tid=0x00007fc784774000 nid=0x1b22 waiting on condition [0x00007fc772bea000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-2-0" #66 prio=5 os_prio=0 tid=0x00007fc784772800 nid=0x1b21 waiting on condition [0x00007fc772ceb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-1-0" #65 prio=5 os_prio=0 tid=0x00007fc784770800 nid=0x1b1f waiting on condition [0x00007fc772dec000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientWorker-OrderedExecutor-0-0" #64 prio=5 os_prio=0 tid=0x00007fc7841a7000 nid=0x1b1e waiting on condition [0x00007fc772eed000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"BookKeeperClientScheduler-OrderedScheduler-0-0" #63 prio=5 os_prio=0 tid=0x00007fc7841a6000 nid=0x1b16 waiting on condition [0x00007fc772fee000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-1-0" #62 prio=5 os_prio=0 tid=0x00007fc77c2c0800 nid=0x1ac6 waiting on condition [0x00007fc7730ef000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-0-0" #61 prio=5 os_prio=0 tid=0x00007fc77c2be800 nid=0x1ac5 waiting on condition [0x00007fc7731f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-15-0" #60 prio=5 os_prio=0 tid=0x00007fc77c2bc800 nid=0x1ac4 waiting on condition [0x00007fc7732f1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-14-0" #59 prio=5 os_prio=0 tid=0x00007fc77c2bb000 nid=0x1ac3 waiting on condition [0x00007fc7733f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-13-0" #58 prio=5 os_prio=0 tid=0x00007fc77c2a3800 nid=0x1ac0 waiting on condition [0x00007fc7734f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-12-0" #57 prio=5 os_prio=0 tid=0x00007fc77c2a1800 nid=0x1abe waiting on condition [0x00007fc7735f4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-11-0" #56 prio=5 os_prio=0 tid=0x00007fc77c29f800 nid=0x1abd waiting on condition [0x00007fc7736f5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-10-0" #55 prio=5 os_prio=0 tid=0x00007fc77c29e000 nid=0x1abc waiting on condition [0x00007fc7737f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-9-0" #54 prio=5 os_prio=0 tid=0x00007fc77c29c000 nid=0x1abb waiting on condition [0x00007fc7738f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-8-0" #53 prio=5 os_prio=0 tid=0x00007fc77c29a000 nid=0x1aba waiting on condition [0x00007fc7739f8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-7-0" #52 prio=5 os_prio=0 tid=0x00007fc77c2b8800 nid=0x1ab9 waiting on condition [0x00007fc773af9000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-6-0" #51 prio=5 os_prio=0 tid=0x00007fc77c2b7000 nid=0x1ab8 waiting on condition [0x00007fc773bfa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-5-0" #50 prio=5 os_prio=0 tid=0x00007fc77c2b5000 nid=0x1ab7 waiting on condition [0x00007fc773cfb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-4-0" #49 prio=5 os_prio=0 tid=0x00007fc77c2b3000 nid=0x1ab6 waiting on condition [0x00007fc773dfc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-3-0" #48 prio=5 os_prio=0 tid=0x00007fc77c2b1800 nid=0x1ab5 waiting on condition [0x00007fc773efd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-2-0" #47 prio=5 os_prio=0 tid=0x00007fc77c2aa800 nid=0x1ab4 waiting on condition [0x00007fc773ffe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-1-0" #46 prio=5 os_prio=0 tid=0x00007fc77c2a9000 nid=0x1ab3 waiting on condition [0x00007fc778183000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-callback-executor-OrderedExecutor-0-0" #45 prio=5 os_prio=0 tid=0x00007fc77c326000 nid=0x1ab2 waiting on condition [0x00007fc778284000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"epollEventLoopGroup-10-1" #43 prio=10 os_prio=0 tid=0x000055bb2a213000 nid=0x19fc runnable [0x00007fc779487000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"pulsar-ordered-OrderedExecutor-3-0-EventThread" #42 daemon prio=5 os_prio=0 tid=0x000055bb2a202000 nid=0x19fa waiting on condition [0x00007fc779588000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-3-0-SendThread(10.33.141.46:2281)" #41 daemon prio=5 os_prio=0 tid=0x000055bb2a200800 nid=0x19f9 waiting on condition [0x00007fc779689000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-1-0" #40 prio=5 os_prio=0 tid=0x00007fc780bb3000 nid=0x19f8 waiting on condition [0x00007fc77978a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-0-0" #39 prio=5 os_prio=0 tid=0x00007fc780baf000 nid=0x19f7 waiting on condition [0x00007fc77988b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-1-0" #38 prio=5 os_prio=0 tid=0x00007fc780b66000 nid=0x19f6 waiting on condition [0x00007fc77998c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"zk-cache-background-OrderedExecutor-0-0" #37 prio=5 os_prio=0 tid=0x00007fc780b64000 nid=0x19f5 waiting on condition [0x00007fc779a8d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-zk-session-watcher-7-1" #36 prio=5 os_prio=0 tid=0x00007fc780b61800 nid=0x19f4 waiting on condition [0x00007fc779b8e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"epollEventLoopGroup-6-1" #35 prio=10 os_prio=0 tid=0x00007fc774302000 nid=0x19f3 runnable [0x00007fc77b090000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.Native.epollWait(Native.java:141) at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

"pulsar-ordered-OrderedExecutor-1-0-EventThread" #34 daemon prio=5 os_prio=0 tid=0x00007fc780b2f800 nid=0x19f2 waiting on condition [0x00007fc77b19d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-1-0-SendThread(10.33.141.44:2281)" #33 daemon prio=5 os_prio=0 tid=0x00007fc780b2f000 nid=0x19f1 waiting on condition [0x00007fc77b29e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-7-0" #32 prio=5 os_prio=0 tid=0x00007fc7840f6000 nid=0x19f0 waiting on condition [0x00007fc77b9f8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-6-0" #31 prio=5 os_prio=0 tid=0x00007fc7840f4800 nid=0x19ef waiting on condition [0x00007fc77baf9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-5-0" #30 prio=5 os_prio=0 tid=0x00007fc7840f3000 nid=0x19ee waiting on condition [0x00007fc77bbfa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-4-0" #29 prio=5 os_prio=0 tid=0x00007fc784428000 nid=0x19ed waiting on condition [0x00007fc77bcfb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-3-0" #28 prio=5 os_prio=0 tid=0x00007fc784426800 nid=0x19ec waiting on condition [0x00007fc77bdfc000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-2-0" #27 prio=5 os_prio=0 tid=0x00007fc784424800 nid=0x19eb waiting on condition [0x00007fc77befd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-1-0" #26 prio=5 os_prio=0 tid=0x00007fc784222000 nid=0x19ea waiting on condition [0x00007fc77bffe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"pulsar-ordered-OrderedExecutor-0-0" #25 prio=5 os_prio=0 tid=0x00007fc78421d000 nid=0x19e9 waiting on condition [0x00007fc7881a7000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"logback-1" #22 daemon prio=5 os_prio=0 tid=0x00007fc7a59cb800 nid=0x19be waiting on condition [0x00007fc788cf9000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"Service Thread" #21 daemon prio=9 os_prio=0 tid=0x00007fc7a55c4800 nid=0x19b7 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #20 daemon prio=9 os_prio=0 tid=0x00007fc7a55b8800 nid=0x19b6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #19 daemon prio=9 os_prio=0 tid=0x00007fc7a55b6800 nid=0x19b5 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #18 daemon prio=9 os_prio=0 tid=0x00007fc7a55b4800 nid=0x19b4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #17 daemon prio=9 os_prio=0 tid=0x00007fc7a55b2800 nid=0x19b3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #16 daemon prio=9 os_prio=0 tid=0x00007fc7a556a800 nid=0x19b2 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #15 daemon prio=9 os_prio=0 tid=0x000055bb29c0b000 nid=0x19b1 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #14 daemon prio=9 os_prio=0 tid=0x00007fc7a5568000 nid=0x19b0 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #13 daemon prio=9 os_prio=0 tid=0x00007fc7a5564800 nid=0x19af waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #12 daemon prio=9 os_prio=0 tid=0x00007fc7a5562000 nid=0x19ae waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #11 daemon prio=9 os_prio=0 tid=0x00007fc7a5591000 nid=0x19ad waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #10 daemon prio=9 os_prio=0 tid=0x00007fc7a558e800 nid=0x19ac waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #9 daemon prio=9 os_prio=0 tid=0x00007fc7a558c800 nid=0x19ab waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" #7 daemon prio=10 os_prio=0 tid=0x00007fc7a5211000 nid=0x19aa runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" #6 daemon prio=10 os_prio=0 tid=0x00007fc7a520d000 nid=0x19a9 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fc7a51f0800 nid=0x19a8 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fc7a51ef000 nid=0x19a7 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc7a51bd000 nid=0x19a6 in Object.wait() [0x00007fc78aaae000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc7a51b8000 nid=0x19a5 in Object.wait() [0x00007fc78abaf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191)

"main" #1 prio=5 os_prio=0 tid=0x000055bb29b78000 nid=0x1941 waiting on condition [0x00007fc7c35bc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method)

"VM Thread" os_prio=0 tid=0x00007fc7a51ae800 nid=0x19a4 runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29bdf800 nid=0x1942 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29be1800 nid=0x1943 runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29be3000 nid=0x1944 runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29be5000 nid=0x1945 runnable

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29be7000 nid=0x1946 runnable

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29be9000 nid=0x1947 runnable

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29bea800 nid=0x1948 runnable

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x000055bb29bec800 nid=0x1949 runnable

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4001000 nid=0x194a runnable

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4003000 nid=0x194b runnable

"Gang worker#10 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4005000 nid=0x194c runnable

"Gang worker#11 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4006800 nid=0x194d runnable

"Gang worker#12 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4008800 nid=0x194e runnable

"Gang worker#13 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a400a800 nid=0x194f runnable

"Gang worker#14 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a400c800 nid=0x1950 runnable

"Gang worker#15 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a400e000 nid=0x1951 runnable

"Gang worker#16 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4010000 nid=0x1952 runnable

"Gang worker#17 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4012000 nid=0x1953 runnable

"Gang worker#18 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4013800 nid=0x1954 runnable

"Gang worker#19 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4015800 nid=0x1955 runnable

"Gang worker#20 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4017800 nid=0x1956 runnable

"Gang worker#21 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4019800 nid=0x1957 runnable

"Gang worker#22 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a401b000 nid=0x1958 runnable

"Gang worker#23 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a401d000 nid=0x1959 runnable

"Gang worker#24 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a401f000 nid=0x195a runnable

"Gang worker#25 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4021000 nid=0x195b runnable

"Gang worker#26 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4022800 nid=0x195c runnable

"Gang worker#27 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4024800 nid=0x195d runnable

"Gang worker#28 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4026800 nid=0x195e runnable

"Gang worker#29 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a4028800 nid=0x195f runnable

"Gang worker#30 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a402a000 nid=0x1960 runnable

"Gang worker#31 (Parallel GC Threads)" os_prio=0 tid=0x00007fc7a402c000 nid=0x1961 runnable

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007fc7a40a6000 nid=0x1983 runnable

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40a8000 nid=0x1984 runnable

"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40a9800 nid=0x1985 runnable

"Gang worker#2 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40ab800 nid=0x1986 runnable

"Gang worker#3 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40ad800 nid=0x1987 runnable

"Gang worker#4 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40af800 nid=0x1988 runnable

"Gang worker#5 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40b1000 nid=0x1989 runnable

"Gang worker#6 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40b3000 nid=0x198a runnable

"Gang worker#7 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40b5000 nid=0x198b runnable

"Gang worker#8 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40b7000 nid=0x198c runnable

"Gang worker#9 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40b8800 nid=0x198d runnable

"Gang worker#10 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40ba800 nid=0x198e runnable

"Gang worker#11 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40bc800 nid=0x198f runnable

"Gang worker#12 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40be800 nid=0x1990 runnable

"Gang worker#13 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c0000 nid=0x1991 runnable

"Gang worker#14 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c2000 nid=0x1992 runnable

"Gang worker#15 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c4000 nid=0x1993 runnable

"Gang worker#16 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c6000 nid=0x1994 runnable

"Gang worker#17 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c7800 nid=0x1995 runnable

"Gang worker#18 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40c9800 nid=0x1996 runnable

"Gang worker#19 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40cb800 nid=0x1997 runnable

"Gang worker#20 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40cd800 nid=0x1998 runnable

"Gang worker#21 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40cf000 nid=0x1999 runnable

"Gang worker#22 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40d1000 nid=0x199a runnable

"Gang worker#23 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40d3000 nid=0x199b runnable

"Gang worker#24 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40d5000 nid=0x199c runnable

"Gang worker#25 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40d6800 nid=0x199d runnable

"Gang worker#26 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40d8800 nid=0x199e runnable

"Gang worker#27 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40da800 nid=0x199f runnable

"Gang worker#28 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40dc800 nid=0x19a0 runnable

"Gang worker#29 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40de000 nid=0x19a1 runnable

"Gang worker#30 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40e0000 nid=0x19a2 runnable

"Gang worker#31 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fc7a40e2000 nid=0x19a3 runnable

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007fc7a406e800 nid=0x1982 runnable

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007fc7a406c800 nid=0x1981 runnable

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007fc7a406a800 nid=0x1980 runnable

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007fc7a4068800 nid=0x197f runnable

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007fc7a4066800 nid=0x197e runnable

"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007fc7a4065000 nid=0x197d runnable

"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007fc7a4063000 nid=0x197c runnable

"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007fc7a4061000 nid=0x197b runnable

"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007fc7a405f000 nid=0x197a runnable

"G1 Concurrent Refinement Thread#9" os_prio=0 tid=0x00007fc7a405d000 nid=0x1979 runnable

"G1 Concurrent Refinement Thread#10" os_prio=0 tid=0x00007fc7a405b800 nid=0x1978 runnable

"G1 Concurrent Refinement Thread#11" os_prio=0 tid=0x00007fc7a4059800 nid=0x1977 runnable

"G1 Concurrent Refinement Thread#12" os_prio=0 tid=0x00007fc7a4057800 nid=0x1976 runnable

"G1 Concurrent Refinement Thread#13" os_prio=0 tid=0x00007fc7a4055800 nid=0x1975 runnable

"G1 Concurrent Refinement Thread#14" os_prio=0 tid=0x00007fc7a4053800 nid=0x1974 runnable

"G1 Concurrent Refinement Thread#15" os_prio=0 tid=0x00007fc7a4052000 nid=0x1973 runnable

"G1 Concurrent Refinement Thread#16" os_prio=0 tid=0x00007fc7a4050000 nid=0x1972 runnable

"G1 Concurrent Refinement Thread#17" os_prio=0 tid=0x00007fc7a404e000 nid=0x1971 runnable

"G1 Concurrent Refinement Thread#18" os_prio=0 tid=0x00007fc7a404c000 nid=0x1970 runnable

"G1 Concurrent Refinement Thread#19" os_prio=0 tid=0x00007fc7a404a800 nid=0x196f runnable

"G1 Concurrent Refinement Thread#20" os_prio=0 tid=0x00007fc7a4048800 nid=0x196e runnable

"G1 Concurrent Refinement Thread#21" os_prio=0 tid=0x00007fc7a4046800 nid=0x196d runnable

"G1 Concurrent Refinement Thread#22" os_prio=0 tid=0x00007fc7a4044800 nid=0x196c runnable

"G1 Concurrent Refinement Thread#23" os_prio=0 tid=0x00007fc7a4042800 nid=0x196b runnable

"G1 Concurrent Refinement Thread#24" os_prio=0 tid=0x00007fc7a4041000 nid=0x196a runnable

"G1 Concurrent Refinement Thread#25" os_prio=0 tid=0x00007fc7a403f000 nid=0x1969 runnable

"G1 Concurrent Refinement Thread#26" os_prio=0 tid=0x00007fc7a403d000 nid=0x1968 runnable

"G1 Concurrent Refinement Thread#27" os_prio=0 tid=0x00007fc7a403b000 nid=0x1967 runnable

"G1 Concurrent Refinement Thread#28" os_prio=0 tid=0x00007fc7a4039000 nid=0x1966 runnable

"G1 Concurrent Refinement Thread#29" os_prio=0 tid=0x00007fc7a4037800 nid=0x1965 runnable

"G1 Concurrent Refinement Thread#30" os_prio=0 tid=0x00007fc7a4035800 nid=0x1964 runnable

"G1 Concurrent Refinement Thread#31" os_prio=0 tid=0x00007fc7a4033800 nid=0x1963 runnable

"G1 Concurrent Refinement Thread#32" os_prio=0 tid=0x00007fc7a4031800 nid=0x1962 runnable

"VM Periodic Task Thread" os_prio=0 tid=0x00007fc7a55c7000 nid=0x19b8 waiting on condition

JNI global references: 14234

codelipenghui commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

tisonkun commented 1 year ago

Closed as stale. Please open a new issue if it's still relevant in maintained versions.

Please don't attach so many logs. No one is willing to debug with such an inline context. You can upload a log file instead.

BTW, I suggest you report KOP related issues on https://github.com/streamnative/kop.