search

sni / mod_gearman

Distribute Naemon Host/Service Checks & Eventhandler with Gearman Queues. Host/Servicegroups affinity included.
http://www.mod-gearman.org
GNU General Public License v3.0
122 stars 42 forks source link

gearmand 0.33 still required to run mod_gearman? #146

Closed dirtyren closed 3 years ago

dirtyren commented 5 years ago

Hello,

sometime ago the recommended gearmand version was a 0.33 patched that is provided on the console labs repos. Should we still use that version or could we run mod_gearman with the epel centos7 provided version 1.1.12? I am running a test machine since Friday and it is running without problems.

Tks.

sni commented 5 years ago

We are still running 0.33 on all our clients. Haven't found the time to test new releases yet. So please let me know if you find anything.

dirtyren commented 5 years ago

Ok @sni , any strange behaviors we find I'll let you know, tks. So far very stable.

gzalo commented 5 years ago

Ok @sni , any strange behaviors we find I'll let you know, tks. So far very stable.

Are you using Nagios or Naemon? we found that with [lib]gearman 1.12 we were having memory leak problems, but after downgrading to 0.33 they went away. If you have a small amount of services or restart nagios periodically it does not matter, but you should monitor its memory just in case.

dirtyren commented 5 years ago

Hey,

Are you using Nagios or Naemon? we found that with [lib]gearman 1.12 we were having memory leak problems, but after downgrading to 0.33 they went away.

naemon, livestatus, mod_gearman and our own broker that uses gearman queues, so far it is working, but the test box has only 86 hosts and 543 services.

If you have a small amount of services or restart nagios periodically it does not matter, but you should monitor its memory just in case. I am going to monitor memory consumption, tks for the tip and soon I will test it all with 25k services.

dirtyren commented 3 years ago

Just to bring this thread up a bit. We are migrating from CentOS 6 to CentOS 8 and trying to use gearmand 1.1.19.

gearmand after one day is at 97% CPU, using naemon, livestatus and mod_gearman-3.3.,

image

gearman_top wont connect image

[root@opdev3 ~]# netstat -anp | grep gearmand | grep "ESTABLISHED" | wc -l 986 I am stressing a little bit running checks each 5s to test all the new software on the new OS.

We used to have gearmand 0.33 at 100% CPU when we had a worker in PHP that would loop for some reason connecting and disconnecting to gearmand. But that is not the case here. Looking on the gearmand github page there are some issues open regarding high CPU usage of gearmand.

Any ideas I could try on this?

[]s.

sni commented 3 years ago

We upgraded gearman 0.33 to 1.x 12 months ago and only had a single issue so far (https://github.com/ConSol/omd/issues/107). The reason was a gearman worker using timeout. PNP in that case.

dirtyren commented 3 years ago

That as caused by a process that was opening connections to gearmand exhausting its available connections. Nothing to do with gearmand, but a behavior with PHP and the gearmand API in objects. It would not disconnect its connection. Related to this comment here https://github.com/gearman/gearmand/issues/241#issuecomment-747598877

Tks.