There are many pinpoint processes in the jstack

[newtypebao]

Recently our application server CPU is always continuous at 30%.We found that there is a tomcat process using CPU continuously,According to the jstack result ,we found that the process has a large number of pinpoint threads running.(Thread.State: RUNNABLE\TIMED_WAITING), I do not know whether it is normal?

here is the jstack info: "commons-pool-EvictionTimer" #45 daemon prio=5 os_prio=0 tid=0x00007f40dc20b000 nid=0x7868 runnable [0x00007f4139648000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque$DescendingItr.(LinkedBlockingDeque.java:1321) at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque$DescendingItr.(LinkedBlockingDeque.java:1321) at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.descendingIterator(LinkedBlockingDeque.java:1183) at org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$EvictionIterator.(BaseGenericObjectPool.java:1144) at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.evict(GenericObjectPool.java:737)

• locked <0x00000000d39c7968> (a java.lang.Object) at org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:1037) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)

"Abandoned connection cleanup thread" #44 daemon prio=5 os_prio=0 tid=0x00007f40dc533800 nid=0x7867 in Object.wait() [0x00007f413994a000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)

• locked <0x00000000c134bd10> (a java.lang.ref.ReferenceQueue$Lock) at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)

"NioBlockingSelector.BlockPoller-2" #41 daemon prio=5 os_prio=0 tid=0x00007f4174e9f000 nid=0x782e runnable [0x00007f4154506000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000000c05876e8> (a sun.nio.ch.Util$3)
• locked <0x00000000c05876d8> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000c05875b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:342)

"NioBlockingSelector.BlockPoller-1" #40 daemon prio=5 os_prio=0 tid=0x00007f4174e8a800 nid=0x782d runnable [0x00007f4154607000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000000c069a110> (a sun.nio.ch.Util$3)
• locked <0x00000000c069a100> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000c0699fd8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:342)

"GC Daemon" #39 daemon prio=2 os_prio=0 tid=0x00007f4174e1d000 nid=0x782c in Object.wait() [0x00007f4155dbc000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method)

• waiting on <0x00000000c0979478> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:117)
• locked <0x00000000c0979478> (a sun.misc.GC$LatencyLock)

"Service Thread" #38 daemon prio=9 os_prio=0 tid=0x00007f4174b30800 nid=0x782a runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #37 daemon prio=9 os_prio=0 tid=0x00007f4174b2e800 nid=0x7829 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #36 daemon prio=9 os_prio=0 tid=0x00007f4174b28800 nid=0x7828 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #35 daemon prio=9 os_prio=0 tid=0x00007f4174b27800 nid=0x7827 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"Pinpoint-stat-monitor(18-0)" #33 daemon prio=5 os_prio=0 tid=0x00007f4174b26000 nid=0x7826 waiting on condition [0x00007f41571e0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0002e30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-agentInfo-sender(17-0)" #32 daemon prio=5 os_prio=0 tid=0x00007f4174b2b800 nid=0x7825 waiting on condition [0x00007f41574e3000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0002378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-UdpStatDataExecutor(16-0)" #31 daemon prio=5 os_prio=0 tid=0x00007f4174ae1000 nid=0x7824 waiting on condition [0x00007f4157dec000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0008840> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.takeOne(AsyncQueueingExecutor.java:127) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.doExecute(AsyncQueueingExecutor.java:94) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.run(AsyncQueueingExecutor.java:79) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-UdpSpanDataExecutor(15-0)" #30 daemon prio=5 os_prio=0 tid=0x00007f4174ae0000 nid=0x7823 waiting on condition [0x00007f41577e6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0009ae0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.takeOne(AsyncQueueingExecutor.java:127) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.doExecute(AsyncQueueingExecutor.java:94) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.run(AsyncQueueingExecutor.java:79) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-TcpDataExecutor(14-0)" #29 daemon prio=5 os_prio=0 tid=0x00007f4174ad4800 nid=0x7822 waiting on condition [0x00007f41579e8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0009d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.takeOne(AsyncQueueingExecutor.java:127) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.doExecute(AsyncQueueingExecutor.java:94) at com.navercorp.pinpoint.profiler.sender.AsyncQueueingExecutor.run(AsyncQueueingExecutor.java:79) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-DataSender-Timer(13-0)" #28 daemon prio=5 os_prio=0 tid=0x00007f4174ada000 nid=0x7821 waiting on condition [0x00007f4157ceb000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-PinpointClientHandler-Timer(12-0)" #27 daemon prio=5 os_prio=0 tid=0x00007f4174acd800 nid=0x7820 waiting on condition [0x00007f41576e5000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-Client-Worker(11-0)" #26 daemon prio=5 os_prio=0 tid=0x00007f4174a93800 nid=0x781f runnable [0x00007f41578e7000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000000c000a188> (a sun.nio.ch.Util$3)
• locked <0x00000000c000a198> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000c000a140> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-Client-Boss(10-0)" #25 daemon prio=5 os_prio=0 tid=0x00007f4174a93000 nid=0x781e runnable [0x00007f4157bea000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000000c0003110> (a sun.nio.ch.Util$3)
• locked <0x00000000c0003120> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000c00030c8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

"Pinpoint-SocketFactory-Timer(9-0)" #24 daemon prio=5 os_prio=0 tid=0x00007f4174a92000 nid=0x781d waiting on condition [0x00007f4157ae9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445) at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at java.lang.Thread.run(Thread.java:748)

"AsyncFileHandlerWriter-1601292138" #22 daemon prio=5 os_prio=0 tid=0x00007f4174718800 nid=0x781c waiting on condition [0x00007f41575e4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000c0008ea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

[koo-taejin]

hi @newtypebao

The number of pinpoint threads is normal.

thanks.

[newtypebao]

hi @koo-taejin Will it continue to occupy the CPU?such as Keep CPU at 30% ？

On 03/26/2019 16:22, koo-taejin wrote:

hi @newtypebao

The number of pinpoint threads is normal.

thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[koo-taejin]

Threads are alived, but threads are not using cpu as you said. If use the cpu extensively, please report with the detailed data.

thanks.

[newtypebao]

Threads are alived, but threads are not using cpu as you said. If use the cpu extensively, please report with the detailed data.

thanks.

@koo-taejin ,The 'detailed data' currently available is those in jstack,what other details do you need?

[koo-taejin]

First, check which thread uses how much cpu. If the pinpoint thread is using a lot of cpu, then please tell me which pinpoint thread is used a lot .

[newtypebao]

First, check which thread uses how much cpu. If the pinpoint thread is using a lot of cpu, then please tell me which pinpoint thread is used a lot .

@koo-taejin ,I have found cpu-consuming threads by referring https://backstage.forgerock.com/knowledge/kb/article/a39551500 And there is no better way for me to detail how much CPU each thread consumes. What is your recommendation for 'check which thread uses how much cpu'?

[koo-taejin]

I recommand you to look at the following article and find out exactly what kind of thread is wrong with it.

https://dzone.com/articles/how-analyze-java-thread-dumps

thanks.