Feature: use this filter to profile/monitor Logstash filter chain

[gquintana]

It would be amazing if we could write something like

filter {
  # Start the stopwatch
  metrics {
    timer_start => [ "filter_%{type}_timer" ]
  }
  # Do some Grok, Date filtering here
  # ...
  # Stop the stopwatch
  metrics {
    timer_end => [ "filter_%{type}_timer" ]
  }
}

I tried adding in the filter something like

    # Start timers
    @timer_start.each do |timer_name|
      timer = event.get("[@metadata][#{timer_name}]")
      if timer == nil
        timer = Metriks::Timer.new(@metric_timers[event.sprintf(timer_name)])
        event.set("[@metadata][#{timer_name}]", timer)
      else
        timer.restart
      end
    end

    # Stop timers
    @timer_start.each do |timer_name|
      timer = event.get("[@metadata][#{timer_name}]")
      if timer!=nil
        timer.stop
      end
    end

Sadly it doesn't work as expected because the Java event doesn't allow to store a Ruby object (Metriks::Time) in it:

  1) LogStash::Filters::Metrics with timer start should add timer
     Failure/Error: Unable to find org.logstash.Valuefier.convertNonCollection(Valuefier.java to read failed line

     Java::JavaLang::IllegalArgumentException:
       Missing Valuefier handling for full class name=org.jruby.RubyObject, simple name=RubyObject
     # org.logstash.Valuefier.convertNonCollection(Valuefier.java:51)
     # org.logstash.Valuefier.convert(Valuefier.java:90)
     # org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:128)
     # org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)
     # org.jruby.internal.runtime.methods.JavaMethod$JavaMethodTwo.call(JavaMethod.java:1126)

[jordansissel]

Can I offer an alternate solution?

Logstash 5.x exposes pipeline metrics for each plugin (especially filter and outputs, right now). This lets you observe time spent in each plugin in your pipeline. Check out this document for more details: https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats

Can you check into this and see if that works for you?

[gquintana]

Wow amazing! Except I have to upgrade from Logstash/ES 2.x :'(

[jordansissel]

@gquintana If it helps, Logstash 5.x works with Elasticsearch 2.x (we test and support this usage)

[jordansissel]

The 5.0+ logstash metrics api shows per-plugin counts and other stats. I think we can resolve this.