Creating a Field from Multiple Default Grok Patterns.

[naisanza]

I'm looking to use multiple grok patterns that, when matched, will be the value for a new field.

For example, here's a log string that I'm working with:

8/19/2014 18:53,6/16/2015 4:21

I want:

"firstSeen" => "8/19/2014 18:53"
"lastSeen" => "6/16/2015 4:21"

Since there isn't an existing grok pattern to match 8/19/2014 18:53 as a timestamp, I can still match it by using individual portions of the existing grok patterns library, such as %{DATE} %{HOUR} %{MINUTE}, but grok doesn't work that way:

grok {
   match =>  [ "message", "%{DATE:firstSeen} %{HOUR:firstSeen}:%{MINUTE:firstSeen}" ]
}

However, I'd like the firstSeen tag to be a concatenation of %{DATE} %{HOUR}:%{MINUTE}

I'm looking to do something like this:

%{(%{DATE} %{HOUR}:%{MINUTE}):firstSeen}

Is that possible?

Edit: I know about the date filter, but there are two timestamps inside the message that needs to be parsed out; and for visualizations, those two fields are needed.

Edit 2: I'm wondering if this will work:

(?<firstSeen>%{DATE} %{HOUR}:%{MINUTE})

Edit 3: I ran the pattern through http://grokconstructor.appspot.com/do/match and it came out matched. This may just work. If it does, a-w-e-s-o-m-e.

[naisanza]

Using (?<firstSeen>%{DATE} %{HOUR}:%{MINUTE}) works!

However, I'm getting _grokparsefailure tags, and out of about 33K events, I'm getting back a handful of 12 or 22 events indexed. I added the grok match pattern to the filter, so I can check it against the message using http://grokdebug.herokuapp.com/, but the pattern is matching the message just fine. Hm, I'm not sure what's causing the parsing issues. I'll have to see if logstash --debug tells me anything.

Edit: Looks like I'm getting a lot of "Failed to flush outgoing items" warnings:

Failed to flush outgoing items {:outgoing_count=>21, :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:464:in `flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in `buffer_flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1507:in `loop'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn, :file=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}

I don't know if this might be an issue with Oracle Java 1.8.0_45 but I'm going to try OpenJKD, again.

root@elk:/home/user# java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.10.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

Edit 2: Still having "Failed to flush outgoing items" warnings

Failed to flush outgoing items {:outgoing_count=>21, :exception=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :backtrace=>["/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:464:in `flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:216:in `buffer_flush'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1507:in `loop'", "/home/user/logstash/elk/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn, :file=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}

Edit 3: The problem seems to be with memory or a grok filter.

[naisanza]

The problem was with using add_field in grok with type => "value". It apparently will throw bulk error messages if you use the field type.