search

vuapo-eth / isf-jclient

java console client for the IOTA Spam Fund (http://iotaspam.com)
MIT License
17 stars 5 forks source link

Thread leak #24

Open rapmue opened 6 years ago

rapmue commented 6 years ago

As soon as there are no synced nodes left or so, the spam client forks a lot of processes. This results in a lot of open threads and in my case the whole server is completely unusable (it's not possible to create new threads).

Attached you'll find some graphs of this behaviour.

image image image image image

Edit: also an excerpt of the log:

[02:56:36] [Logger] [INF] >>> PERFORMANCE REPORT >>> PoW: 01.36s | Efficiency: 92.52% | GetTips: 00.63s | TipPool: 6/9 | Heap: 251/7122/91 MB [02:56:38] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions) [02:56:39] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions) [02:56:42] [NodeMngr] [WRN] no api available, waiting ... [02:56:43] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions) [02:56:48] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions) [02:56:57] [NodeMngr] [WRN] no api available, waiting ... [02:56:59] [TimeAbrt] [WRN] action 'create spam transaction' took too long and was aborted after 30 seconds [02:57:12] [NodeMngr] [WRN] no api available, waiting ... [02:57:13] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions)

and in the end before reboot:

[06:23:36] [Logger] [INF] >>> PERFORMANCE REPORT >>> PoW: 01.36s | Efficiency: 84.75% | GetTips: 00.63s | TipPool: 6/9 | Heap: 268/7122/197 MB [06:23:37] [TimeAbrt] [ERR] the program run out of memory and could not create a new thread [06:23:36] [NodeMngr] [WRN] no api available, waiting ... [06:23:37] [TimeAbrt] [DBG] Heap: 268/7122/197 MB [06:23:38] [TimeAbrt] [DBG] Threads: M8/P0/A10704/C2/T5/N5/S0 [06:23:38] [TimeAbrt] [ERR] unable to create new native thread [06:23:39] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions) [06:23:41] [TimeAbrt] [ERR] the program run out of memory and could not create a new thread [06:23:41] [TimeAbrt] [WRN] action 'broadcasting transaction' took too long and was aborted after 10 seconds (this message only shows up after 10 abortions)

vuapo-eth commented 6 years ago

Thanks for the detailed report. Hopefully commit https://github.com/mikrohash/isf-jclient/commit/afcbee08690969a699503f866da5593c88a9bf34 (dev branch) will fix that. Running the modified version right now so we will find out the next time this happens.

vuapo-eth commented 6 years ago

looks like https://github.com/mikrohash/isf-jclient/commit/afcbee08690969a699503f866da5593c88a9bf34 makes it even worse. will try something different

rapmue commented 6 years ago

Hm, I let the released version run, and it was stable until last night around midnight. Just to let you know, if your version run through.

vuapo-eth commented 6 years ago

Second try: https://github.com/mikrohash/isf-jclient/commit/c5cefc59a1abdf9588ddb46c44c55c5d931f706f

Compiled jar: here - will print the thread count of the critical threads (B&C) into your terminal like this: [20:04:56] [Logger] [INF] A6/B0/C1/D0/E0

vuapo-eth commented 6 years ago

that did not fix it

ghost commented 5 years ago

I have seen similar issue, in my case it looks like it depletes all the file descriptors on the system, so you cannot run any command in the terminal, the machine is more or less dead. This happens on 3 different Ubuntu machines that I am using, after been running for a while, the Mac does not seem to have this problem.

ghost commented 5 years ago

[20:39:42] [TimeAbrt] [ERR] unable to create new native thread

java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at isf.logic.TimeAbortCall.call(TimeAbortCall.java:39) at isf.spam.NodeManager.connectToNode(NodeManager.java:110) at isf.spam.NodeManager.access$200(NodeManager.java:24) at isf.spam.NodeManager$3.run(NodeManager.java:128)

[20:39:41] [TimeAbrt] [DBG] Heap: 244/3641/170 MB [20:39:42] [TimeAbrt] [ERR] the program run out of memory and could not create a new thread [20:39:41] [TimeAbrt] [DBG] Heap: 244/3641/170 MB [20:39:42] [TimeAbrt] [ERR] the program run out of memory and could not create a new thread [20:39:41] [TimeAbrt] [DBG] Heap: 244/3641/170 MB [20:39:41] [TimeAbrt] [DBG] Heap: 244/3641/170 MB