search

carlosCharz / fcmxmppserverv2

XMPP Connection Server for FCM using the latest version of the Smack library (4.3.4) + Connection Draining Implementation
47 stars 33 forks source link

High CPU usage (99%) and thread count (20+) #4

Closed tannerhallman closed 7 years ago

tannerhallman commented 7 years ago

Is anyone else having an issue with a high CPU and thread count that just appeared (yesterday evening.) This is happening on my localhost (macintosh) and my appserver (linux).

I pulled a heap dump, but I can't make sense of it. Is this just happening to me?

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):

"pool-5-thread-2" #22 prio=5 os_prio=31 tid=0x00007fd489ba7800 nid=0xd717 waiting on condition [0x0000700001bed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000795d5b728> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"pool-5-thread-1" #21 prio=5 os_prio=31 tid=0x00007fd488b96800 nid=0xd50f waiting on condition [0x0000700001aea000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000795d5b728> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"FirebaseDatabaseWorker" #20 daemon prio=5 os_prio=31 tid=0x00007fd48a279800 nid=0xd30f waiting on condition [0x00007000019e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007958998b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Smack-Single Threaded Executor 0 (0)" #19 daemon prio=5 os_prio=31 tid=0x00007fd489039000 nid=0xd107 waiting on condition [0x00007000018e4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000740587fc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Smack-Incoming Processor 0 (0)" #17 daemon prio=5 os_prio=31 tid=0x00007fd48a27a800 nid=0xcd0f waiting on condition [0x00007000016de000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000740588378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Smack Packet Reader (0)" #15 daemon prio=5 os_prio=31 tid=0x00007fd48925c000 nid=0xcb03 runnable [0x00007000015da000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
    - locked <0x0000000740219b60> (a java.lang.Object)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    - locked <0x0000000740219c18> (a sun.security.ssl.AppInputStream)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    - locked <0x000000074021bcb8> (a java.io.InputStreamReader)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.read1(BufferedReader.java:212)
    at java.io.BufferedReader.read(BufferedReader.java:286)
    - locked <0x000000074021bcb8> (a java.io.InputStreamReader)
    at org.jivesoftware.smack.util.ObservableReader.read(ObservableReader.java:41)
    at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:2992)
    at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
    at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1384)
    at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.parsePackets(XMPPTCPConnection.java:1177)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader.access$300(XMPPTCPConnection.java:956)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketReader$1.run(XMPPTCPConnection.java:971)
    at java.lang.Thread.run(Thread.java:745)

"Smack Packet Writer (0)" #14 daemon prio=5 os_prio=31 tid=0x00007fd489206800 nid=0xc90f waiting on condition [0x00007000014d8000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000074021d8d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.jivesoftware.smack.util.ArrayBlockingQueueWithShutdown.take(ArrayBlockingQueueWithShutdown.java:289)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.nextStreamElement(XMPPTCPConnection.java:1312)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.writePackets(XMPPTCPConnection.java:1329)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter.access$2900(XMPPTCPConnection.java:1192)
    at org.jivesoftware.smack.tcp.XMPPTCPConnection$PacketWriter$1.run(XMPPTCPConnection.java:1240)
    at java.lang.Thread.run(Thread.java:745)

"AppKit Thread" #11 daemon prio=5 os_prio=31 tid=0x00007fd489119800 nid=0xa0b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Ctrl-Break" #9 daemon prio=5 os_prio=31 tid=0x00007fd489078800 nid=0x4d03 runnable [0x0000700001149000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:79)
    at java.lang.Thread.run(Thread.java:745)

"Service Thread" #8 daemon prio=9 os_prio=31 tid=0x00007fd48901f000 nid=0x4903 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007fd48901c000 nid=0x4703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007fd489001000 nid=0x4503 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007fd488844800 nid=0x4303 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fd489828800 nid=0x360f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fd48a009800 nid=0x2e03 in Object.wait() [0x000070000092e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x000000074028ffb8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
    - locked <0x000000074028ffb8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fd489816800 nid=0x2c03 in Object.wait() [0x000070000082b000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000740224b48> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
    - locked <0x0000000740224b48> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=31 tid=0x00007fd488804000 nid=0x1003 runnable [0x0000700000182000]
   java.lang.Thread.State: RUNNABLE
    at com.wedevol.xmpp.EntryPoint.main(EntryPoint.java:56)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

"VM Thread" os_prio=31 tid=0x00007fd489813800 nid=0x2a03 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fd48a003800 nid=0x2203 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fd48880d800 nid=0x2403 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fd488818800 nid=0x2603 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fd488819000 nid=0x2803 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007fd488844000 nid=0x4b03 waiting on condition 

JNI global references: 1778

Heap
 PSYoungGen      total 38400K, used 16617K [0x0000000795580000, 0x000000079a080000, 0x00000007c0000000)
  eden space 33280K, 49% used [0x0000000795580000,0x00000007965ba418,0x0000000797600000)
  from space 5120K, 0% used [0x0000000797600000,0x0000000797600000,0x0000000797b00000)
  to   space 5120K, 0% used [0x0000000799b80000,0x0000000799b80000,0x000000079a080000)
 ParOldGen       total 62976K, used 6983K [0x0000000740000000, 0x0000000743d80000, 0x0000000795580000)
  object space 62976K, 11% used [0x0000000740000000,0x00000007406d1fc8,0x0000000743d80000)
 Metaspace       used 22316K, capacity 22618K, committed 23040K, reserved 1069056K
  class space    used 2829K, capacity 2950K, committed 3072K, reserved 1048576K
carlosCharz commented 7 years ago

Do not use the reconnect implementation. It has a while true. I guess that is the problem.