logstash-plugins / logstash-filter-multiline

Apache License 2.0
18 stars 16 forks source link

Unable to pattern match two consecutive newlines #8

Open jsvd opened 9 years ago

jsvd commented 9 years ago

Migrated from : https://logstash.jira.com/browse/LOGSTASH-631

Rails production.log has two empty lines in between each event. My multiline filter with pattern "\n\n" is not matching and I know for sure I have two newlines.
Here is the filter I tried which isn't working with a typical rails production.log:
multiline
{ type => "rails_prod_log" pattern => "\n\n" negate => "true" what => "previous" }
The following works but isn't quite what I want:
multiline
{ type => "rails_prod_log" pattern => "^$" negate => "true" what => "previous" }
The reason:
Sometimes an event/stacktrace has a broken single empty line with one newline character in it, in between the stacktrace. I wouldn't want to split that stacktrace into two separate events so I need to match on two newlines (ie "\n\n")
I'd appreciate a work-around for now until version 1.1.2 comes out.
Thanks!
Navid

example log:

Started GET "/assets/logo.png" for 127.0.0.1 at 2012-10-14 16:31:02 -0700
Served asset /logo.png - 304 Not Modified (0ms)

Started GET "/" for 127.0.0.1 at 2012-10-15 21:38:47 -0700
Processing by HomeController#index as HTML
  MOPED: 127.0.0.1:27017 COMMAND      database=admin command={:ismaster=>1} (0.0004ms)
  MOPED: 127.0.0.1:27017 QUERY        database=mongoid collection=users selector={"_id"=>{"$in"=>["23542722234aaa0001"]}} flags=[:slave_ok] limit=0 skip=0 fields=nil (0.0009ms)
Filter chain halted as :authentication_prep rendered or redirected
Completed 401 Unauthorized in 3ms

Started GET "/" for 127.0.0.1 at 2012-10-16 13:54:40 -0700
Processing by HomeController#index as HTML
  MOPED: 127.0.0.1:27017 COMMAND      database=admin command={:ismaster=>1} (0.0004ms)
  MOPED: 127.0.0.1:27017 QUERY        database=mongoid collection=users selector={"_id"=>{"$in"=>["23542722234aaa0001"]}} flags=[:slave_ok] limit=0 skip=0 fields=nil (0.0005ms)
Filter chain halted as :authentication_prep rendered or redirected
Completed 401 Unauthorized in 2ms
wiibaa commented 9 years ago

Potentially related to https://github.com/elasticsearch/logstash/issues/2262 and https://github.com/elasticsearch/logstash/issues/2262#issuecomment-67448930