search

sni / mod_gearman

Distribute Naemon Host/Service Checks & Eventhandler with Gearman Queues. Host/Servicegroups affinity included.
http://www.mod-gearman.org
GNU General Public License v3.0
122 stars 42 forks source link

Event handlers, uniq and /dev/urandom #102

Closed adrianlzt closed 8 years ago

adrianlzt commented 8 years ago

Where are having troubles with Icinga+mod_gearman when using reload. After each reload, a new file handler to /dev/urandom appears, till max files open limit is reached and breaks icinga.

The trouble comes when mod_gearman tries to submit a new event handler job to Gearman. It doesn't pass a uniq value, so libuuid is used to generate a new one.

With old versions of uuid (v2.17.2, the one that comes with RHEL 6.x), it opens a static fd to /dev/urandom.

When Icinga reloads, it does a dlclose() to mod_gearman shared lib, this clear the static variable storing the /dev/urandom fd, but doesn't close the fd. So next time an event handler is fired, a new fd to /dev/urandom is open.

Upgrading libuuid seems to solve the problem, but there is not upgrade for RHEL 6.x, and it is used by others packages (syslinux among them).

Other possibility will be to pass a uniq identifier to each new event handler job, maybe something like host+srv+time_ns.

sni commented 8 years ago

Thats awful. How often does icinga crash? I mean, i do not tend to implement a workaround for a issue which is already fixed in newer libuuid releases.

adrianlzt commented 8 years ago

We have changed reload for restart to workaround the issue. I hope at least this description could help someone with the same trouble.

sni commented 8 years ago

Indeed, thanks for the detailed explaination.