[ Messages not matched by a Grok pattern are not dropped ]

[jerome83136]

Hello,

I'm using Grok custom patterns to match my logs and output them in files

My config looks like this:

input {
 file {
  start_index => 0
  path => '/central_logs/input/prod/webservers/zp2web0?/apache/prospect/access_FH?_log'
  type => 'prospect'
 }
 file {
  start_index => 0
  path => '/central_logs/input/prod/webservers/zp2web0?/apache/webshop/access_*MALE_log'
  type => 'webshop'
 }
}

filter {
 grok {
  extra_patterns_file => '/conf/logstash/patterns.grok.prospect'
  match => '%{IP:clientip} \- \- \[%{TIMESTAMP}\]\s*\"%{METHOD}\s*\/webshop\/%{P0X:p0x}\/'
 }

output {
 if [p0x] != 'p02' {
  file {
   path => "/central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log"
  }
 }
}

The Grok's patterns file:

IP (?:[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})
USER_HTTP (?:.*)
USER_LYRECO (?:.*)
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
TIMESTAMP (?:[0-9]{2}/%{MONTH}/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2} [+-][0-9]{4})
METHOD (?:\b\w+\b)
APPLICATION (?:(webshop))
P0X (?:(P0[1,3-9]))
REQUEST (?:%{APPLICATION}\/(?:(?:%{P0X}|.*)).*)

It works. I have several output files created, with the correct name. But I get a number of the following errors when I launch node-logstash:

[...]
[Thu, 21 Apr 2016 10:33:40 GMT] ERROR [output_file] output to file /central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log still failing.
[Thu, 21 Apr 2016 10:33:42 GMT] ERROR [output_file] output to file /central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log still failing.
[...]

I think some messages does not match the Grok patterns, so they don't have the ${p0x} field populated; preventing node-logstash from writing to the correct output file.

Am I right ? If yes, is there a way to drop non-matching messages ? (I want only the matching messages to be processed)

Many thanks for your help

Best regards Jérôme

[jerome83136]

Hello,

Update:

It seems that if I make the configuration like bellow it works:

output {
 if [p0x] == 'p01' {
  file {
   idle_timeout => 600
   path => '/central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log'
  }
 }
 if [p0x] == 'p03' {
  file {
   idle_timeout => 600
   path => '/central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log'
  }
 }
 if [p0x] == 'p04' {
  file {
   idle_timeout => 600
   path => '/central_logs/output/prod/webservers/#{type}/#{type}_#{p0x}.#{now:YYYYMMDD}.log'
  }
 }

[...]

Best regards Jérôme

[bpaquet]

Please see https://github.com/bpaquet/node-logstash/commit/a8e0b4bce96198ac14a801f17ccab96c0f48e629.

I just added tags and fields management. This can be a cleaner solution to solve your initial problem.

[jerome83136]

Just great ! Thank you very much