search

netfishers-onl / Netshot

Network Configuration and Compliance Management
http://www.netfishers.onl/netshot
240 stars 57 forks source link

v0.19.0 is not reaping threads #275

Closed anli77 closed 8 months ago

anli77 commented 10 months ago

Hi,

First off, thanks for a very good device configuration handling system, but after updating to v19.0 it seems that the change to the new JVM appears to have some side effects, it is not reaping threads consistently, slowly leaving threads until the whatever max value is set is reached, on the docker container we have setup it is 2048.

Netshot v0.18.3a with the exact same configuration file and have "netshot.tasks.threadcount = 6" set the JVM hovers around 92-116 threads active, reaping dead ones correctly as thread ids are increasing but concurrent threads are stable in the range mentioned.

Netshot v0.19.0 with the same "netshot.tasks.threadcount = 6" setting will consume all 2048 concurrent threads in about 4 hours on our setup when pushed to collect all active devices, about 1600, setting "netshot.tasks.threadcount = 1" will slow down the leaking since less are started concurrently, but it will eventually reach 2048 and then no further collection can be done.

Already running threads, such as the one serving the REST API etc keep working as expected, but any new threads will not start.

A note, not all thread types are leaked, as when checking active thread count will decrease with one or two, then increase again, some testing points to the compliance checks being the thread types that are not reaped, in our case we use both simple text and python compliance rules, no java script ones, but not being a java developer I could be wrong.

SCadilhac commented 10 months ago

Hello, thanks for the feedback. It's likely to be related to Python scripts only. This fix should help: 724c1b92f773bd93927c8379f4cbcbc3075bb53a Could you confirm?

anli77 commented 10 months ago

Hi, I would love to test but both the "build from source" wiki page and Docker files for a build seems use the old JVM and versions, so is there update instructions somewhere?

SCadilhac commented 10 months ago

The instructions are still correct, but the JDK version must be aligned with installation instructions indeed. I'm switching to automated builds in parallel.

SCadilhac commented 8 months ago

Should be fixed in 0.19.2.