Closed naisanza closed 9 years ago
Using (?<firstSeen>%{DATE} %{HOUR}:%{MINUTE})
works!
However, I'm getting _grokparsefailure
tags, and out of about 33K events, I'm getting back a handful of 12 or 22 events indexed. I added the grok match
pattern to the filter, so I can check it against the message
using http://grokdebug.herokuapp.com/, but the pattern
is matching the message
just fine. Hm, I'm not sure what's causing the parsing issues. I'll have to see if logstash --debug
tells me anything.
Edit: Looks like I'm getting a lot of "Failed to flush outgoing items" warnings:
Failed to flush outgoing items {:outgoing_count=>21, :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:464:in `flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in `buffer_flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1507:in `loop'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn, :file=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}
I don't know if this might be an issue with Oracle Java 1.8.0_45
but I'm going to try OpenJKD, again.
root@elk:/home/user# java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.10.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
Edit 2: Still having "Failed to flush outgoing items" warnings
Failed to flush outgoing items {:outgoing_count=>21, :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:464:in `flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in `buffer_flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1507:in `loop'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn, :file=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}
Edit 3: The problem seems to be with memory or a grok filter.
The problem was with using add_field
in grok with type => "value"
. It apparently will throw bulk
error messages if you use the field type
.
I'm looking to use multiple grok patterns that, when matched, will be the value for a new field.
For example, here's a log string that I'm working with:
I want:
Since there isn't an existing grok pattern to match
8/19/2014 18:53
as a timestamp, I can still match it by using individual portions of the existing grok patterns library, such as%{DATE}
%{HOUR}
%{MINUTE}
, but grok doesn't work that way:However, I'd like the
firstSeen
tag to be a concatenation of%{DATE} %{HOUR}:%{MINUTE}
I'm looking to do something like this:
Is that possible?
Edit: I know about the
date
filter, but there are two timestamps inside themessage
that needs to be parsed out; and for visualizations, those two fields are needed.Edit 2: I'm wondering if this will work:
Edit 3: I ran the pattern through http://grokconstructor.appspot.com/do/match and it came out matched. This may just work. If it does, a-w-e-s-o-m-e.