iotaledger / iri

IOTA Reference Implementation
Other
1.15k stars 370 forks source link

java.lang.Thread.State: WAITING (parking) #394

Open MClarkDev opened 6 years ago

MClarkDev commented 6 years ago

The details are a bit vague, but my node has stopped responding. It seems to still be accepting data, but all threads seem to be deadlocked; nothing is coming back out.

[root@iota centos]# cat /proc/loadavg
0.16 0.08 0.21 1/223 13797

[root@iota centos]# cat /proc/meminfo | grep Mem
MemTotal:        7816112 kB
MemFree:          537880 kB
MemAvailable:    1291764 kB

[root@iota centos]# ps aux|grep java|grep -v grep
root      7655 87.0 71.9 7265760 5620308 ?     Ssl  Dec02 7465:13 /bin/java -Xms4g -Xmx4g -jar /opt/iota/bin/iri.latest.jar -c /opt/iota/bin/iota.ini

[root@iota centos]# netstat -lpn|grep 14265
tcp6     129      0 :::14265                :::*                    LISTEN      7655/java

[root@iota centos]# time curl localhost:14265
♥

real    0m47.190s
user    0m0.004s
sys     0m0.011s

I don't have jstack, and did not launch with debug enabled. Here is the output of kill -3 7655

threads.txt

They seem to all be in the same state:

 "XNIO-1 task-19" #1943 prio=5 os_prio=0 tid=0x00007f936402ebd0 nid=0x967 waiting on condition [0x00007f93407ad000]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x00000006c4d592e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

dead

kaushlendra-rai commented 6 years ago

@MClarkDev I'm facing the same issue for my service.

Have you been able to resolve or at least figure out the root cause of the problem? Some direction would be helpful.