search

sensu-plugins / sensu-plugins-logs

This plugin provides native instrumentation for monitoring log files or system logs via journald for regular expressions, and a Sensu handler for logging Sensu events to log files.
http://sensu-plugins.io
MIT License
8 stars 31 forks source link

A flag to ignore traversing a log file on initial read #10

Open codepattern opened 8 years ago

codepattern commented 8 years ago

First, thank you for the ruby plugin.

I would like to suggest the capability to ignore the initial search on a log file. This is option is important when you are starting to watch an existing log file which in some cases are extremely large in size. The initial read takes a higher that normal amount of cpu and memory to traverse the file. This new option would give the check-log.rb the capability to commerce monitoring from the last line / end of file position line in the file.

codepattern commented 8 years ago

Any update or thoughts on this request? Thanks

aayore commented 8 years ago

I can see the downside to reading a huge log file. But this could also be solved with logrotate or other system utilities, so I wouldn't think it's a terribly high priority. Just my $.02.

codepattern commented 8 years ago

Thanks for the response. But logrotate would not solve the issue. If I wanted to start monitoring the log file from the present point in time and the log is halfway through its size limit rotation, the check would still have to traverse the previous content. What I am asking is that it not read anything previous to its start or give the ability to set this flag on or off.

majormoses commented 7 years ago

@codepattern how would this work? I am wondering if say a max number of lines might be what we are looking for?

majormoses commented 7 years ago

@codepattern I have not heard from you on how this would work and would need some additional detail on how you imagine this feature being used. Without such here are some thoughts, apologies if I am missing about your intended use case that makes this more desireable.

What I am asking is that it not read anything previous to its start

Are you referring to the first time the check is invoked and a pos file is created? So the behavior would be if pos file does not exist then write out the pos file as if it read and evaluated it with an ok? If such behavior was to be considered it would certainly need to be behind an option with a default of false.

Log events are inherently in the past so I fail to understand the value of choosing to not evaluate all past events. From a practical standpoint this feature will ignore real events that should be evaluated; while we can empower the user to do so I don't think it's a good decision and question the choices that got them to this point.

In the meantime while we discuss this you could try these workarounds:

  1. set occurrences to > 1 which should always ensure that it needs to fail more than once before alerting with whatever handler you are using. https://sensuapp.org/docs/0.28/reference/plugins.html#check-definition-attributes
  2. Resolve the first occurrence using an api or uchiwa dashboard
  3. mv or rm the existing log file and recreate it.