search

logstash-plugins / logstash-input-cloudwatch

A Logstash input to pull events from the Amazon Web Services CloudWatch API
Apache License 2.0
43 stars 27 forks source link

Multiple filters does not work. #46

Closed hendryanw closed 5 years ago

hendryanw commented 5 years ago

V1:

input {
  # ElasticSearch AZ1
  cloudwatch {
    type => "custom-metrics"
    namespace => "CWAgent"
    metrics => [ "disk_used_percent", "disk_free", "disk_used", "disk_total" ]
    filters => [ 
        { path => "/" },
        { InstanceId => "i-0348f41427efbe150" },
        { device => "nvme0n1p1" },
        { fstype => "ext4" }
    ]
    region => "ap-southeast-1"
    aws_credentials_file => "/etc/logstash/conf.d/aws_credentials_file"
  }
}
output {
    stdout { codec => rubydebug }
}

This will output an errors: 

[ERROR] 2018-09-28 08:36:12.816 [[main]<cloudwatch] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::CloudWatch aws_credentials_file=>"/etc/logstash/conf.d/aws_credentials_file", namespace=>"CWAgent", metrics=>["disk_used_percent", "disk_free", "disk_used", "disk_total"], filters=>[{"path"=>"/"}, {"InstanceId"=>"i-0348f41427efbe150"}, {"device"=>"nvme0n1p1"}, {"fstype"=>"ext4"}], id=>"795992bdd431e8496ae8d2faf950f5e656fd817606706f92699bad7184acd473", type=>"custom-metrics", region=>"ap-southeast-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_6dfad351-a845-4e8a-8ff1-a3400dc757ce", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", statistics=>["SampleCount", "Average", "Minimum", "Maximum", "Sum"], interval=>900, period=>300, combined=>false>
  Error: undefined method `each_pair' for #<Array:0x16180ab5>
Did you mean?  each_entry
  Exception: NoMethodError
  Stack: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch-2.2.2/lib/logstash/inputs/cloudwatch.rb:175:in `from_resources'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch-2.2.2/lib/logstash/inputs/cloudwatch.rb:161:in `block in run'
org/jruby/RubyArray.java:1734:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch-2.2.2/lib/logstash/inputs/cloudwatch.rb:155:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-cloudwatch-2.2.2/lib/logstash/inputs/cloudwatch.rb:149:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:408:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:402:in `block in start_input'

V2: 

input {
  # ElasticSearch AZ1
  cloudwatch {
    type => "custom-metrics"
    namespace => "CWAgent"
    metrics => [ "disk_used_percent", "disk_free", "disk_used", "disk_total" ]
    filters => {
        path => "/"
        InstanceId => "i-0348f41427efbe150"
        device => "nvme0n1p1"
        fstype => "ext4"
    }
    region => "ap-southeast-1"
    aws_credentials_file => "/etc/logstash/conf.d/aws_credentials_file"
  }
}
output {
    stdout { codec => rubydebug }
}

This will instead query the metric using each filter, the plugin will initiate 4 API request for one metric, but we will need them to be queried in a single request, otherwise it wouldn't work.

[INFO ] 2018-09-28 08:39:46.143 [[main]<cloudwatch] cloudwatch - Polling CloudWatch API
[INFO ] 2018-09-28 08:39:46.158 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2018-09-28 08:39:46.818 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-09-28 08:39:52.931 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 4.485635 0 retries] list_metrics(namespace:"CWAgent")

[INFO ] 2018-09-28 08:39:53.579 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.361178 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_total",start_time:2018-09-28 08:24:52 UTC,end_time:2018-09-28 08:39:52 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"path",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.659 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.054871 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_total",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"InstanceId",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.692 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.025595 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_total",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"device",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.730 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.032873 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_total",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"fstype",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.781 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.03495 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"path",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.898 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.107968 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"InstanceId",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.931 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.027331 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"device",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:53.974 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.033452 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"fstype",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.009 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.025037 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_free",start_time:2018-09-28 08:24:53 UTC,end_time:2018-09-28 08:39:53 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"path",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.033 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.018691 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_free",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"InstanceId",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.131 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.087238 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_free",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"device",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.168 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.02437 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_free",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"fstype",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.210 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.025173 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used_percent",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"path",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.258 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.036867 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used_percent",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"InstanceId",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.294 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.020601 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used_percent",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"device",value:"[FILTERED]"}])

[INFO ] 2018-09-28 08:39:54.387 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 0.080744 0 retries] get_metric_statistics(namespace:"CWAgent",metric_name:"disk_used_percent",start_time:2018-09-28 08:24:54 UTC,end_time:2018-09-28 08:39:54 UTC,period:300,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"fstype",value:"[FILTERED]"}])

V3:

input {
  # ElasticSearch AZ1
  cloudwatch {
    type => "custom-metrics"
    namespace => "CWAgent"
    metrics => [ "disk_used_percent", "disk_free", "disk_used", "disk_total" ]
    filters => {
        path => "/",
        InstanceId => "i-0348f41427efbe150",
        device => "nvme0n1p1",
        fstype => "ext4"
    }
    region => "ap-southeast-1"
    aws_credentials_file => "/etc/logstash/conf.d/aws_credentials_file"
  }
}
output {
    stdout { codec => rubydebug }
}

This will output an error on configuration:

[ERROR] 2018-09-28 08:43:12.050 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 8, column 20 (byte 217) after input {\n  # ElasticSearch AZ1\n  cloudwatch {\n    type => \"custom-metrics\"\n    namespace => \"CWAgent\"\n    metrics => [ \"disk_used_percent\", \"disk_free\", \"disk_used\", \"disk_total\" ]\n    filters => {\n        path => \"/\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:157:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:in `block in converge_state'"]}

Can you please let me know if I am doing something wrong, or is it really an issue? The documentation also does not specify how would you use multiple filters for this. It just specify the type as Array which in my case didn't work.

Thanks! Hendry

hendryanw commented 5 years ago

I have updated the issue's description with more detailed information.

hendryanw commented 5 years ago

After reading the source, I figure out that it was because I did not specified the combined property to true.

The following configuration works:

input {
  # ElasticSearch AZ1
  cloudwatch {
    type => "custom-metrics"
    namespace => "CWAgent"
    metrics => [ "disk_used_percent", "disk_free", "disk_used", "disk_total" ]
    filters => { 
        path => "/"
        InstanceId => "i-0348f41427efbe150"
        device => "nvme0n1p1"
        fstype => "ext4"
    }
    combined => true
    region => "ap-southeast-1"
    aws_credentials_file => "/etc/logstash/conf.d/aws_credentials_file"
  }
}
output {
    stdout { codec => rubydebug }
}

I still don't understand the reason why combined property is set to false, but I hope this will helped others in the future.