search

matteocorti / nagios_plugins

18 stars 7 forks source link

Issue with more then one session, when a check_mk request is sent at exactly the same time #123

Closed matteocorti closed 8 years ago

matteocorti commented 8 years ago

Original reporter: _joopboonen@web.de

Sometimes "no updates available" are reported when more then one person is logged into icinga, with an active status interface.

I've checked the data that is sent as a request of check_mk via this entry /usr/lib64/nagios/plugins/check_updates -t 60 --yum-arguments="--disablerepo=..." 2>&1 | tee -a /tmp/mrpe_check_updates.out in /etc/check_mk/mrpe.cfg

When it fails due to 2 sessions I see:

rpmdb: Thread/process 1540/140597775955872 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
CHECK_UPDATES OK - no updates available | total_updates=0;0;0
CHECK_UPDATES CRITICAL - 7 non-critical updates available | total_updates=7;0;0 security_updates=0;0;0
coreutils.x86_64
coreutils-libs.x86_64
kernel.x86_64
kernel-firmware.noarch
liblog4shib1.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch

At this time I see this error.
rpmdb: Thread/process 1540/140597775955872 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

Which is only seen when two sessions are running at nearly the same, but they don't always show "CHECK_UPDATES OK - no updates available | total_updates=0;0;0"

See:

rpmdb: Thread/process 31565/139827728320416 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
Existing lock /var/run/yum.pid: another copy is running as pid 31572.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory :  22 M RSS (219 MB VSZ)
    Started: Fri May 24 16:03:55 2013 - 00:01 ago
    State  : Running, pid: 31572
CHECK_UPDATES CRITICAL - 7 non-critical updates available | total_updates=7;0;0 security_updates=0;0;0
coreutils.x86_64
coreutils-libs.x86_64
kernel.x86_64
kernel-firmware.noarch
liblog4shib1.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch
CHECK_UPDATES CRITICAL - 7 non-critical updates available | total_updates=7;0;0 security_updates=0;0;0
coreutils.x86_64
coreutils-libs.x86_64
kernel.x86_64
kernel-firmware.noarch
liblog4shib1.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch
Existing lock /var/run/yum.pid: another copy is running as pid 31812.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory :  22 M RSS (219 MB VSZ)
    Started: Fri May 24 16:04:55 2013 - 00:01 ago
    State  : Running, pid: 31812

The database failure is something that normally hasn't been seen when the requests are not fired at nearly the same time:

CHECK_UPDATES CRITICAL - 7 non-critical updates available | total_updates=7;0;0 security_updates=0;0;0
coreutils.x86_64
coreutils-libs.x86_64
kernel.x86_64
kernel-firmware.noarch
liblog4shib1.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch
CHECK_UPDATES CRITICAL - 7 non-critical updates available | total_updates=7;0;0 security_updates=0;0;0
coreutils.x86_64
coreutils-libs.x86_64
kernel.x86_64
kernel-firmware.noarch
liblog4shib1.x86_64
selinux-policy.noarch
selinux-policy-targeted.noarch
Existing lock /var/run/yum.pid: another copy is running as pid 31332.
Another app is currently holding the yum lock; waiting for it to exit...
  The other application is: yum
    Memory :  21 M RSS (219 MB VSZ)
    Started: Fri May 24 16:02:55 2013 - 00:01 ago
    State  : Running, pid: 31332

Somehow yum doesn't recognise on time that the rpmdb is locked. (This shows Error: rpmdb open failed and CHECK_UPDATES OK - no updates available | total_updates=0;0;0)

I don't know what the best way would be to solve this. Probably to not sent any/or the latest CHECK_UPDATES values or to wait for some time as long an check_update instance is running?

matteocorti commented 8 years ago

This issue was moved to matteocorti/check_updates#3