lausser / check_logfiles

A plugin (monitoring-plugin, not nagios-plugin, see also http://is.gd/PP1330) which scans logfiles for patterns.
https://omd.consol.de/docs/plugins/check_logfiles/
GNU General Public License v2.0
46 stars 27 forks source link

Critical pattern found but plugin exits OK #68

Closed Napsty closed 2 years ago

Napsty commented 2 years ago

Hi Gerhard

Version:

$ /usr/lib/nagios/plugins/check_logfiles -V
check_logfiles v4.0.1.3

Config file:

#$seekfilesdir = '/var/tmp';
$seekfilesdir = '/etc/nagios';
# where the state information will be saved.

$protocolsdir = '/var/tmp';
# where protocols with found patterns will be stored.

$scriptpath = '/usr/lib/nagios/plugins';
# where scripts will be searched for.

@searches = (
  {
    tag => 'logstash',
    logfile => '/var/log/logstash/logstash-plain.log',
    criticalpatterns => ['ERROR', 'Pipeline started'],
    options => 'noprotocol,nocount,sticky,nosavethresholdcount,nosavestate,allyoucaneat'
  }
);

Note: I set the $seekfilesdir on purpose to a non-writeable directory because all the options were ignored and that was the only way the plugin would read all lines of the log file.

Script runs with this config file, finds matched pattern (Pipeline started) but instead of showing a CRITICAL output and exit, the plugin returns OK:

$ /usr/lib/nagios/plugins/check_logfiles -f /etc/nagios/logfiles-xxx.conf -v
Fri Feb 25 08:00:07 2022: ==================== /var/log/logstash/logstash-plain.log ==================
Fri Feb 25 08:00:07 2022: try pre2seekfile /etc/nagios/logfiles-xxx.logstash-plain.log.logstash instead
Fri Feb 25 08:00:07 2022: try pre3seekfile /tmp/logfiles-xxx._var_log_logstash_logstash-plain.log.logstash instead
Fri Feb 25 08:00:07 2022: no seekfile /etc/nagios/logfiles-xxx._var_log_logstash_logstash-plain.log.logstash found
Fri Feb 25 08:00:07 2022: but logfile /var/log/logstash/logstash-plain.log found
Fri Feb 25 08:00:07 2022: eat all you can
Fri Feb 25 08:00:07 2022: ILS lastlogfile = /var/log/logstash/logstash-plain.log
Fri Feb 25 08:00:07 2022: ILS lastoffset = 0 / lasttime = 0 (Thu Jan  1 01:00:00 1970) / inode = 66305:262625
Fri Feb 25 08:00:07 2022: the logfile grew to 5010
Fri Feb 25 08:00:07 2022: opened logfile /var/log/logstash/logstash-plain.log
Fri Feb 25 08:00:07 2022: logfile /var/log/logstash/logstash-plain.log (modified Wed Jan 19 08:58:31 2022 / accessed Thu Feb 24 15:20:43 2022 / inode 262625 / inode changed Wed Jan 19 08:58:31 2022)
Fri Feb 25 08:00:07 2022: relevant files: logstash-plain.log
Fri Feb 25 08:00:07 2022: moving to position 0 in /var/log/logstash/logstash-plain.log
Fri Feb 25 08:00:07 2022: MATCH CRITICAL Pipeline started with [2022-01-19T07:41:22,202][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
Fri Feb 25 08:00:07 2022: MATCH CRITICAL Pipeline started with [2022-01-19T08:58:31,240][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
Fri Feb 25 08:00:07 2022: stopped reading at position 5010
Fri Feb 25 08:00:07 2022: no sticky error from last run
OK - no errors or warnings|'logstash_lines'=30 'logstash_warnings'=0 'logstash_criticals'=2 'logstash_unknowns'=0

What am I doing wrong? I can't find the needle.

thx

Napsty commented 2 years ago

I should start writing an issue immediately before starting my research next time, because whenever I do this, I find the solution in the next 5 minutes... ¯_(ツ)_/¯

Problem was the nocount option:

Controls whether hits are counted and decide over the final exit code.

All good :-)