search

alibaba / Sentinel

A powerful flow control component enabling reliability, resilience and monitoring for microservices. (面向云原生微服务的高可用流控防护组件)
https://sentinelguard.io/
Apache License 2.0
22.4k stars 8.03k forks source link

Dashboard large memory footprint (AtomicReference), cause full GC and surge of CPU usage #1520

Open Tank-zhu opened 4 years ago

Tank-zhu commented 4 years ago

背景：

sentinel 版本1.7.0

描述：

当前应用上线3个月以上，于昨天下午19：00出 现老年代不 释放 导致频繁FULL GC CPU 飙升

image

导出的dump文件内发现 AtomicReference对象占用达到90%

image

线程信息如下


"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#15
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#24
    at java.lang.Thread.run(Thread.java:745)

"sentinel-command-center-executor-thread-1" prio=5 tid=14 RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
       Local Variable: java.net.SocksSocketImpl#1
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
       Local Variable: java.net.SocksSocketImpl#2
    at java.net.ServerSocket.accept(ServerSocket.java:513)
       Local Variable: java.net.Socket#1
       Local Variable: java.net.ServerSocket#1
    at com.alibaba.csp.sentinel.transport.command.SimpleHttpCommandCenter$ServerThread.run(SimpleHttpCommandCenter.java:188)
       Local Variable: com.alibaba.csp.sentinel.transport.command.SimpleHttpCommandCenter$ServerThread#1
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#30
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.ThreadPoolExecutor#1
       Local Variable: java.util.concurrent.FutureTask#29
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#1
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-26" daemon prio=5 tid=313 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#23
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#51
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#21
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 12" prio=5 tid=34 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#9
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#9
       Local Variable: sun.nio.ch.Util$2#9
       Local Variable: java.util.Collections$UnmodifiableSet#216
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#5
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#5
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-10" daemon prio=5 tid=50 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#48
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#4
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#2
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-6" prio=5 tid=71 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#64
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#25
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 3" prio=5 tid=24 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#18
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#18
       Local Variable: sun.nio.ch.Util$2#18
       Local Variable: java.util.Collections$UnmodifiableSet#236
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#14
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#14
    at java.lang.Thread.run(Thread.java:745)

"logback-2" daemon prio=5 tid=139 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#71
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#34
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 11" prio=5 tid=33 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#10
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#10
       Local Variable: java.util.Collections$UnmodifiableSet#217
       Local Variable: sun.nio.ch.Util$2#10
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#6
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#6
    at java.lang.Thread.run(Thread.java:745)

"ContainerBackgroundProcessor[StandardEngine[Tomcat]]" daemon prio=5 tid=19 TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1356)
       Local Variable: org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor#1
       Local Variable: java.lang.String#88830
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-ClientPoller-0" daemon prio=5 tid=51 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#2
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#2
       Local Variable: sun.nio.ch.Util$2#2
       Local Variable: java.util.Collections$UnmodifiableSet#209
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
       Local Variable: org.apache.tomcat.util.net.NioEndpoint$Poller#2
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-4" daemon prio=5 tid=44 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#19
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#5
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#3
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-21" daemon prio=5 tid=308 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#44
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#48
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#18
    at java.lang.Thread.run(Thread.java:745)

"sentinel-system-status-record-task-thread-1" daemon prio=5 tid=59 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#8
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#93
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#4
       Local Variable: java.util.concurrent.locks.ReentrantLock#1570
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#4
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#16
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-5" prio=5 tid=70 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#63
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#26
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 13" prio=5 tid=35 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#7
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#7
       Local Variable: java.util.Collections$UnmodifiableSet#214
       Local Variable: sun.nio.ch.Util$2#7
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#4
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#4
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-15" daemon prio=5 tid=302 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#49
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#42
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#12
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-12" daemon prio=5 tid=62 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#34
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#14
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#10
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-5" daemon prio=5 tid=45 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#28
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#6
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#4
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-4" prio=5 tid=69 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#61
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#27
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-24" daemon prio=5 tid=311 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#24
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#53
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#23
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-23" daemon prio=5 tid=310 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#50
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#50
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#20
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 6" prio=5 tid=27 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#15
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#15
       Local Variable: sun.nio.ch.Util$2#15
       Local Variable: java.util.Collections$UnmodifiableSet#225
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#11
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#11
    at java.lang.Thread.run(Thread.java:745)

"sentinel-heartbeat-send-task-thread-1" daemon prio=5 tid=12 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#59
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#5
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
       Local Variable: java.util.concurrent.locks.ReentrantLock#1318
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#1
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#1
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#2
    at java.lang.Thread.run(Thread.java:745)

"DestroyJavaVM" prio=5 tid=56 RUNNABLE

"sentinel-dashboard-metrics-fetchWorker-thread-3" prio=5 tid=74 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
       Local Variable: java.util.concurrent.CountDownLatch$Sync#1
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#56
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:250)
       Local Variable: java.util.concurrent.ConcurrentHashMap#2
       Local Variable: com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher#1
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
       Local Variable: java.lang.String#105171
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run(<unknown string>)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#1
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.FutureTask#1
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#22
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-AsyncTimeout" daemon prio=5 tid=54 TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1149)
       Local Variable: org.apache.coyote.AbstractProtocol$AsyncTimeout#1
    at java.lang.Thread.run(Thread.java:745)

"container-0" prio=5 tid=20 TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
       Local Variable: org.apache.catalina.core.StandardServer#1
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:182)

"Reference Handler" daemon prio=10 tid=2 WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)

"http-nio-18088-ClientPoller-1" daemon prio=5 tid=52 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#1
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#1
       Local Variable: sun.nio.ch.Util$2#1
       Local Variable: java.util.Collections$UnmodifiableSet#208
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
       Local Variable: org.apache.tomcat.util.net.NioEndpoint$Poller#1
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 4" prio=5 tid=25 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#17
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#17
       Local Variable: sun.nio.ch.Util$2#17
       Local Variable: java.util.Collections$UnmodifiableSet#235
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#13
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#13
    at java.lang.Thread.run(Thread.java:745)

"logback-6" daemon prio=5 tid=183 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#70
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#38
    at java.lang.Thread.run(Thread.java:745)

"Finalizer" daemon prio=8 tid=3 WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
       Local Variable: java.lang.ref.ReferenceQueue#18
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
       Local Variable: java.lang.System$2#1

"sentinel-dashboard-metrics-fetchService-thread-8" prio=5 tid=73 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#62
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#23
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-1" daemon prio=5 tid=41 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#80
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#25
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
       Local Variable: java.util.concurrent.locks.ReentrantLock#1527
       Local Variable: java.util.concurrent.atomic.AtomicInteger#42
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
       Local Variable: org.apache.tomcat.util.threads.TaskQueue#1
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: org.apache.tomcat.util.threads.ThreadPoolExecutor#1
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#3
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#1
    at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" prio=5 tid=21 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#21
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#21
       Local Variable: sun.nio.ch.Util$2#21
       Local Variable: java.util.Collections$UnmodifiableSet#283
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
       Local Variable: org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor#2
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
       Local Variable: org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager#2
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
       Local Variable: org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1#2
       Local Variable: org.apache.http.impl.nio.client.InternalIODispatch#2
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 14" prio=5 tid=36 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#6
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#6
       Local Variable: sun.nio.ch.Util$2#6
       Local Variable: java.util.Collections$UnmodifiableSet#213
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#3
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#3
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-14" daemon prio=5 tid=301 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#20
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#11
    at java.lang.Thread.run(Thread.java:745)

"Signal Dispatcher" daemon prio=9 tid=4 RUNNABLE

"http-nio-18088-exec-22" daemon prio=5 tid=309 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#37
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#49
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#19
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 5" prio=5 tid=26 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#16
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#16
       Local Variable: sun.nio.ch.Util$2#16
       Local Variable: java.util.Collections$UnmodifiableSet#234
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#12
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#12
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 10" prio=5 tid=32 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#11
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.Util$2#11
       Local Variable: sun.nio.ch.EPollSelectorImpl#11
       Local Variable: java.util.Collections$UnmodifiableSet#218
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#7
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#7
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 1" prio=5 tid=22 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#20
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#20
       Local Variable: sun.nio.ch.Util$2#20
       Local Variable: java.util.Collections$UnmodifiableSet#238
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#16
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#16
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-6" prio=5 tid=77 RUNNABLE
    at org.apache.http.message.AbstractHttpMessage.addHeader(AbstractHttpMessage.java:104)
       Local Variable: java.lang.String#91058
       Local Variable: java.lang.String#92315
    at org.apache.http.protocol.RequestUserAgent.process(RequestUserAgent.java:76)
    at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
       Local Variable: org.apache.http.client.methods.HttpRequestWrapper#6
       Local Variable: org.apache.http.client.protocol.HttpClientContext#6
       Local Variable: org.apache.http.HttpRequestInterceptor[]#1
    at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
    at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128)
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
       Local Variable: org.apache.http.concurrent.BasicFuture#6
       Local Variable: org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl#6
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:214)
       Local Variable: java.util.HashMap$KeyIterator#15
       Local Variable: java.util.concurrent.atomic.AtomicLong#11679
       Local Variable: java.util.concurrent.atomic.AtomicLong#11678
       Local Variable: java.util.concurrent.CountDownLatch#4
       Local Variable: java.util.concurrent.atomic.AtomicLong#11680
       Local Variable: java.util.concurrent.ConcurrentHashMap#13
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
       Local Variable: java.lang.String#304549
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run(<unknown string>)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#24
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.FutureTask#24
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#19
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-9" daemon prio=5 tid=49 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#46
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#10
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#8
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-1" prio=5 tid=65 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#86
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#42
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
       Local Variable: java.util.concurrent.locks.ReentrantLock#1534
       Local Variable: java.util.concurrent.ArrayBlockingQueue#3
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ThreadPoolExecutor#9
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#31
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 7" prio=5 tid=28 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#14
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#14
       Local Variable: java.util.Collections$UnmodifiableSet#224
       Local Variable: sun.nio.ch.Util$2#14
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#10
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#10
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-17" daemon prio=5 tid=304 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#26
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#43
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#13
    at java.lang.Thread.run(Thread.java:745)

"logback-1" daemon prio=5 tid=97 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#95
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#78
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#5
       Local Variable: java.util.concurrent.locks.ReentrantLock#2011
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#5
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#33
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 16" prio=5 tid=38 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#4
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#4
       Local Variable: java.util.Collections$UnmodifiableSet#211
       Local Variable: sun.nio.ch.Util$2#4
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#1
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#1
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 9" prio=5 tid=31 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#12
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#12
       Local Variable: java.util.Collections$UnmodifiableSet#219
       Local Variable: sun.nio.ch.Util$2#12
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#8
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#8
    at java.lang.Thread.run(Thread.java:745)

"logback-7" daemon prio=5 tid=212 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#72
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#39
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-5" prio=5 tid=76 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#33
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#20
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-16" daemon prio=5 tid=303 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#18
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#44
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#14
    at java.lang.Thread.run(Thread.java:745)

"logback-4" daemon prio=5 tid=156 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#73
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#36
    at java.lang.Thread.run(Thread.java:745)

"sentinel-metrics-record-task-thread-1" daemon prio=5 tid=60 RUNNABLE
    at com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric.fromBucket(ArrayMetric.java:191)
       Local Variable: com.alibaba.csp.sentinel.slots.statistic.base.WindowWrap#6539
    at com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric.details(ArrayMetric.java:165)
       Local Variable: java.util.ArrayList#78
       Local Variable: com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric#320
    at com.alibaba.csp.sentinel.node.StatisticNode.metrics(StatisticNode.java:121)
       Local Variable: java.util.concurrent.ConcurrentHashMap#30
    at com.alibaba.csp.sentinel.node.metric.MetricTimerListener.run(MetricTimerListener.java:44)
       Local Variable: java.util.TreeMap#1
       Local Variable: com.alibaba.csp.sentinel.node.metric.MetricTimerListener#1
       Local Variable: com.alibaba.csp.sentinel.node.ClusterNode#80
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#33
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#5
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#3
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#15
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-1" prio=5 tid=64 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#59
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#83
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
       Local Variable: java.util.concurrent.ArrayBlockingQueue#2
       Local Variable: java.util.concurrent.locks.ReentrantLock#1532
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ThreadPoolExecutor#8
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#32
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-Acceptor-0" daemon prio=5 tid=53 RUNNABLE
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
       Local Variable: sun.nio.ch.ServerSocketChannelImpl#1
       Local Variable: java.lang.Object#14269
       Local Variable: java.net.InetSocketAddress[]#5
       Local Variable: java.io.FileDescriptor#34
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:482)
       Local Variable: org.apache.tomcat.util.net.NioEndpoint$Acceptor#1
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-20" daemon prio=5 tid=307 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#22
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#47
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#17
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-13" daemon prio=5 tid=63 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#35
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#13
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#9
    at java.lang.Thread.run(Thread.java:745)

"sentinel-heartbeat-send-task-thread-2" daemon prio=5 tid=57 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#6
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#11
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 15" prio=5 tid=37 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#5
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#5
       Local Variable: java.util.Collections$UnmodifiableSet#212
       Local Variable: sun.nio.ch.Util$2#5
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#2
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#2
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-4" prio=5 tid=75 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#54
       Local Variable: java.util.concurrent.CountDownLatch$Sync#2
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:250)
       Local Variable: java.util.concurrent.ConcurrentHashMap#5
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
       Local Variable: java.lang.String#129459
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run(<unknown string>)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#2
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.FutureTask#2
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#21
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-25" daemon prio=5 tid=312 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#51
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#52
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#22
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-8" daemon prio=5 tid=48 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#36
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#5
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-8" prio=5 tid=79 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#43
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#17
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 2" prio=5 tid=23 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#19
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#19
       Local Variable: sun.nio.ch.Util$2#19
       Local Variable: java.util.Collections$UnmodifiableSet#237
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#15
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#15
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-3" prio=5 tid=68 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#60
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#28
    at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 8" prio=5 tid=29 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#13
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#13
       Local Variable: sun.nio.ch.Util$2#13
       Local Variable: java.util.Collections$UnmodifiableSet#223
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
       Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#9
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
       Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#9
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-19" daemon prio=5 tid=306 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#21
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#46
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#16
    at java.lang.Thread.run(Thread.java:745)

"logback-8" daemon prio=5 tid=213 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#77
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#40
    at java.lang.Thread.run(Thread.java:745)

"NioBlockingSelector.BlockPoller-1" daemon prio=5 tid=40 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#3
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#3
       Local Variable: sun.nio.ch.Util$2#3
       Local Variable: java.util.Collections$UnmodifiableSet#210
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)

"http-nio-18088-exec-18" daemon prio=5 tid=305 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#47
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#45
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#15
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-2" prio=5 tid=67 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#41
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#29
    at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-1" prio=5 tid=30 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
       Local Variable: sun.nio.ch.EPollArrayWrapper#8
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
       Local Variable: sun.nio.ch.EPollSelectorImpl#8
       Local Variable: java.util.Collections$UnmodifiableSet#215
       Local Variable: sun.nio.ch.Util$2#8
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
       Local Variable: org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor#1
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
       Local Variable: org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager#1
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
       Local Variable: org.apache.http.impl.nio.client.InternalIODispatch#1
       Local Variable: org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1#1
    at java.lang.Thread.run(Thread.java:745)

"logback-3" daemon prio=5 tid=151 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#1
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#35
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-6" daemon prio=5 tid=46 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#45
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#8
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#6
    at java.lang.Thread.run(Thread.java:745)

"sentinel-time-tick-thread" daemon prio=5 tid=58 TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at java.lang.Thread.sleep(Thread.java:340)
    at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
    at com.alibaba.csp.sentinel.util.TimeUtil$1.run(TimeUtil.java:37)
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetch-task-thread-1" prio=5 tid=39 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#89
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#58
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
       Local Variable: java.util.concurrent.locks.ReentrantLock#1536
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#2
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#2
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#12
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-7" prio=5 tid=78 RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
    at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
    at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
       Local Variable: java.lang.Object#14295
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:231)
       Local Variable: org.apache.http.impl.nio.reactor.SessionRequestImpl#1
    at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:445)
       Local Variable: org.apache.http.nio.pool.AbstractNIOConnPool$2#23
    at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:277)
       Local Variable: org.apache.http.nio.pool.LeaseRequest#1
       Local Variable: org.apache.http.concurrent.BasicFuture#8
       Local Variable: org.apache.http.impl.nio.conn.CPool#1
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:268)
       Local Variable: org.apache.http.concurrent.BasicFuture#7
    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
       Local Variable: org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl#5
       Local Variable: org.apache.http.concurrent.BasicFuture#5
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:214)
       Local Variable: java.util.HashMap$KeyIterator#14
       Local Variable: java.util.concurrent.atomic.AtomicLong#11677
       Local Variable: java.util.concurrent.atomic.AtomicLong#11676
       Local Variable: java.util.concurrent.atomic.AtomicLong#11675
       Local Variable: java.util.concurrent.CountDownLatch#3
       Local Variable: java.util.concurrent.ConcurrentHashMap#12
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
       Local Variable: java.lang.String#105189
    at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run(<unknown string>)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       Local Variable: java.util.concurrent.Executors$RunnableAdapter#23
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       Local Variable: java.util.concurrent.FutureTask#23
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#18
    at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-2" daemon prio=5 tid=42 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#39
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
    at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#7
    at java.lang.Thread.run(Thread.java:745)

"Attach Listener" daemon prio=9 tid=329 RUNNABLE

"logback-5" daemon prio=5 tid=164 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#69
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#37
    at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-2" prio=5 tid=66 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#17
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#30
    at java.lang.Thread.run(Thread.java:745)`
Tank-zhu commented 4 years ago

有没有大佬碰到过呢，可以帮忙看看吗？

draco1023 commented 2 years ago

也遇到了同样的问题，版本也是1.7.0，CPU持续在600以上，dump堆栈的文件有将近12G，都没办法使用mat进行分析，使用 jmap -histo:live 得到的文件里看到的内容也是 AtomicReference 超高

 num     #instances         #bytes  class name
----------------------------------------------
   1:     231586675     3705386800  java.util.concurrent.atomic.AtomicReference
   2:       1807504      954362112  [Ljava.util.concurrent.atomic.AtomicReference;
   3:       2794980      233120104  [C
   4:       1866532      134390304  com.alibaba.csp.sentinel.dashboard.datasource.entity.MetricEntity
   5:       4067051       97609224  java.util.concurrent.atomic.AtomicLong
   6:       2808151       67395624  java.util.Date
   7:       2793113       67034712  java.lang.String
   8:       1866435       59725920  com.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8$Node
   9:       1866435       59725920  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
  10:       1866696       44800704  java.lang.Long
  11:       1866435       44794440  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
  12:        225938       40104352  [Lcom.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8$Node;
  13:        451876       21690048  [Ljava.util.concurrent.atomic.AtomicLong;
  14:        225938       21690048  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
  15:        226029       18080856  [J
  16:        225938       16267536  com.googlecode.concurrentlinkedhashmap.ConcurrentHashMapV8
  17:        225938       10845024  [[Ljava.util.concurrent.atomic.AtomicReference;
  18:        363717        8729208  java.util.concurrent.ConcurrentLinkedQueue$Node
  19:        259987        8319584  java.util.concurrent.ConcurrentHashMap$Node
  20:          5696        8070688  [B