Generate one event per metric name

[jamesharr]

I'd like to use the metrics plugin to generate 1 even per metric name. Clone can be used to manually split apart an event generated by metrics, however it's not always possible to know the name of every metric ahead of time. With 1 event per metric name, a slightly different event format would be useful where the event name is stored in a field instead of in a field-name. See below for examples.

Examples of dynamic metrics: Firewall rule, country, domain (webserver), URL, log event source server, etc.

Example logstash configuration:

filter {
  if "firewall" in [tag] {
    metrics {
      meter => [ "firewall.%{firewall_rule}" ]
      add_tag => "metric"
      rates => [ 1 ]
      flush_interval => 60
      clear_interval => 3600
      split_events => true
    }
  }
}

Example output from metrics plugin

{
@timestamp => "2016-02-21T15:04:28.414Z"
name => "firewall.rule1"
rate_1m => 0.29
count => 8
tag => [ "metric" ]
}
{
@timestamp => "2016-02-21T15:04:28.414Z"
name => "firewall.rule2"
rate_1m => 1
count => 60
tag => [ "metric" ]
}

Example output usage:

output {
  if "metric" in [tag] { 

    # Store each metric reading in graphite (or some other time series database)
    graphite {
      metrics => [ "%{name}.count_1m", "%{rate_1m}" ]
    }

    # Store each metric reading in an ES index that's kept for a long period of time (years)
    elasticsearch {
      index => [ "metrics" ]
    }

  } else {
    # Store normal events in an ES database that's purged on a shorter timeframe (1mo/1wk)
    elasticsearch {
      index => "logstash-%{+YYYY.MM.dd}"
    }
  }
}

[berglh]

I have a similar requirement to produce metrics on a per event source basis. I have raised this issue for discussion in the logstash discussion forum and didn't see this particular enhancement discussion.

Trying to Count and Graph Multiple Event Sources

The concise version is that I'm trying to do this:

filter {
  if [message] {
    metrics {
      meter => [ "%{host}" ]
      add_tag => "metric"
    }
  }
}
output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
    stdout {
      codec => rubydebug
    }
  }
}

With this valid configuration, the metrics filter does a great job at counting each message based on the event source host and producing the metrics, just not in a usable fashion when it comes to graphing time series.

I end up with metric events that contain an undetermined number of source hosts, I have no way to reliably filter the 1000+ possible server names to produce the individual metric events. If each event source host had it's own metrics event with common metric fields, very similar to the use case of the OP, I could easily produce charts on events per second per source host to a specific logstash instance.

This is particularly useful for determining what server is flooding my Redis queues with a large volume of events. I am currently doing this with statsd output plugin, however, it's pretty unreliable sending increment (1 event) stats and having statsd aggregate them and store in InfluxDB. It works, but it makes a lot more sense to have logstash calculate the rates and to store them in an elasticsearch metrics cluster.

Another slight niggle is that when a host stops sending events, the rate_1m never reaches zero, even after one minute it still producing a value over 0 which indicates to me there is an error with the way that value is being calculated although this is probably a gripe for separate plugin issue.

[DEvil0000]

+1 I need this feature to get a structure I can use 'term split' for graphs later (kibana/grafana)

[DEvil0000]

BTW: this is not exactly the same as #17. In #17 the data is still structured as split_%{key} = { <values> } here it is flat and the name/key is a value pared with key name.

name = %{key}
<values>

[Battleroid]

Is there any traction on this? I'm running into a similar problem where I can create the metrics for a slew of dynamic names based on a field, but I cannot send them anywhere (in particular statsd) since I cannot reference them.

Similar to @berglh's issue, I'd like to track a running list of dynamic metrics to pinpoint specific hosts/indices in our pipeline for issues.

[solatis]

I have the same problem; wouldn't mind just using an additional script or something that does the cloning and splitting, but at the moment I cannot use the generated metrics events in a usable fashion for e.g. Kibana.

[chbuescher]

I've built a patch to metrics.rb, that introduces a new boolean option emit_single_metrics. If set to false - the default - it behaves as before, but if set to true, events are generated in the proposed flat format with one metric per event.

metric.rb.diff.txt

[berglh]

@chbuescher That's awesome, I've built the patch into the latest version of logstash-filter-metrics and going to test it shortly, thanks.

[elvarb]

You can work around this by using Telegraf https://github.com/influxdata/telegraf

Along with sending the output to Elasticsearch, also send the event to Telegraf, which in turn can handle it like a proper metric event, aggregate and send out to InfluxDB or some other of its outputs.

[berglh]

@elvarb What receiver on Telegraf do you use? I've been actually been using the statsd plugin; and that's fine but it's pretty un-refined way of collecting statistics (spamming UDP packets for each Logstash event).

Can you elaborate on how Telegraf can split the various metric instances in a given event from a standard logstash-filter-metrics event? Thanks :+1:

[elvarb]

@berglh I used just the standard tcp input plugin in telegraf, or if I used the statsd plugin. Don't have access to the system I set this up on anymore.

It basically meant that I was doubling the data coming from the logstash machine, one flow went to elasticsearch and the other to telegraf. Another benefit I got from this is that I selected which fields were sent over to telegraf so there I could use influxdb's tag system. So for an event that included a sourceip field that became a tag set in Influxdb, then I could filter in on only events for a specific sourceip value.

The logstash output I used was just plain tcp, I did not use the metric filter.

[viranch]

Here's a work-around that I have used for this use-case:

Manipulate the metric events to put metric values in a list using the ruby filter (while recording the metric names in the list), and then use the split filter to generate individual events for each metric name in the list:

filter {
  if "firewall" in [tags] {
    metrics {
      meter => [ "firewall.%{firewall_rule}" ]
      add_tag => "metric"
      rates => [ 1 ]
      flush_interval => 60
      clear_interval => 3600
    }
  }

  if "metric" in [tags] {
    ruby {
      code => "
        metrics = []
        event.to_hash.each_key do |k|
          if k.start_with? 'firewall.'
            values = event.remove(k)
            values['name'] = k
            metrics << values
          end
        end
        event.set('metrics', metrics)
      "
    }

    split {
      field => "metrics"
      target => "metric"
      remove_field => "metrics"
      add_tag => "metric"
    }
  }
}

With this, we get roughly the following output events:

{
@timestamp => "2016-02-21T15:04:28.414Z"
metric => {
  name => "firewall.rule1"
  rate_1m => 0.29
  count => 8
}
tags => [ "metric" ]
}
{
@timestamp => "2016-02-21T15:04:28.414Z"
metric => {
  name => "firewall.rule2"
  rate_1m => 1
  count => 60
}
tags => [ "metric" ]
}

[1vish99]

Will this feature be added ?

[1vish99]

@viranch, did you face any performance issues with the work around.? We will be having around 50000+ events to split

[berglh]

@1tarak I've been using this fork and it's still installing OK in the most recent logstash versions (6.7.1 for me): https://github.com/berglh/logstash-filter-metrics

Assuming you are using logstash in the default path:

# Prepare Ruby Build Environmet
export GEM_HOME=/usr/share/logstash/vendor/bundle/jruby/2.3.0
export JRUBY_HOME=/usr/share/logstash/vendor/jruby/bin
export PATH=$(echo $PATH):/usr/share/logstash/vendor/jruby/bin:/usr/share/logstash/vendor/bundle/jruby/2.3.0/bin
/usr/share/logstash/bin/logstash-plugin remove logstash-filter-metrics
cd /usr/share/logstash/plugins
git clone https://github.com/berglh/logstash-filter-metrics.git
cd /usr/share/logstash/plugins/logstash-filter-metrics
gem build logstash-filter-metrics.gemspec
../../bin/logstash-plugin install *.gem

Then you can do stuff like this:

    metrics {
      meter => [ "%{[@metadata][pipeline]}" ]
      rates => 1
      emit_single_metrics => true
      add_field => { 
        "[@metadata][metric]" => "true"
        "[@metadata][ls-dev-mode]" => "${LS_DEV_MODE:true}"
        "[metric][description]" => "logstash_processor_pipeline_throughput"
        "[service]" => "logstash"
        "[type]" => "metrics"
        "index" => "uq-metrics-%{+YYYY.MM.dd}"
      }
    }
    metrics {
      meter => [ "service" ]
      rates => 1    
      emit_single_metrics => true
      add_field => {
        "[@metadata][metric]" => "true"
        "[@metadata][ls-dev-mode]" => "${LS_DEV_MODE:true}"
        "[service]" => "logstash"
        "[type]" => "metrics"
        "[metric][description]" => "logstash_processor_total_throughput"
        "index" => "uq-metrics-%{+YYYY.MM.dd}"
      }
    }

[berglh]

@viranch, did you face any performance issues with the work around.? We will be having around 50000+ events to split

With that version of the metrics plugin I'm indexing about 4,100,000 metrics a day to a single Elasticsearch node metrics instance.

[1vish99]

@berglh, Thank you! i will install and try.

[berglh]

@1tarak As I'm using this in production, if it's not working in Logstash 7 let me know and I'll merge in the latest changes from the upstream plugin so it should build properly there as well as I'll need to do this anyway after the logstash-filter-dns bug is fixed :)

[1vish99]

I am still using LS 6.2.3 in our prod. So we are good for now. Thank you!

[voiprodrigo]

@berglh Can you please rebase so it bumps to version 4.0.6? I'm using Logstash 6.8.1 and would like to try your patch. Thanks!

[voiprodrigo]

@berglh I'm using your patch based on 4.0.5 on 6.8.1 and it's working fine, thanks!

[voiprodrigo]

Is this working with Logstash 7?

[berglh]

@voiprodrigo I was using 7.9.1 and it was working fine, I'm just building 7.16.1 right now, I don't think this should be a problem though.

[berglh]

Ok, I've rebased for logstash 7.16.1. The primary branch has updated to main to reflect the upstream repo, and the patch is applied on main and on the tag v4.0.7. Patch applied fine, I'm assuming this will work.

Edit: I can confirm the multi-metrics patch is working on 7.16.1.

[voiprodrigo]

Thanks for the quick update! I'm running without issue on 7.16.1 as well.