Collect metric data from namespace AWS/ES

[tankhuu]

I can get the metric data with this AWS CLI command.

But I can't configure the logstash input to collect it, Because the dimensions need 2 parameters, which are DomainName, and ClientId to collect data for namespace AWS/ES. But when I configure in logstash, They had been separated to 2 polls command. Please help. Here are my configuration file, and the error I receive:

Configuration File:

input {
  cloudwatch {
    namespace => "AWS/ES"
    metrics => [ "CPUUtilization" ]
    filters => {
      "DomainName" => "my-ES-domainName"
      "ClientId" => "my-AWS-AccountId"
    }
    aws_credentials_file => "/etc/logstash/aws_credential.yml"
    region => "ap-southeast-1"
  }
}

output {
  if [namespace] == "AWS/ES" {
    stdout {codec => rubydebug}
  }
}

Running Log Received:

[2017-09-27T09:01:59,531][INFO ][logstash.inputs.cloudwatch] [AWS CloudWatch 200 1.549 0 retries] list_metrics(:namespace=>"AWS/ES")

[2017-09-27T09:01:59,533][INFO ][logstash.inputs.cloudwatch] Polling metric CPUUtilization
[2017-09-27T09:01:59,534][INFO ][logstash.inputs.cloudwatch] Filters: [{:name=>"DomainName", :values=>["my-ES-domainName"]}, {:name=>"ClientId", :values=>["my-AWS-AccountId"]}]
[2017-09-27T09:01:59,534][INFO ][logstash.inputs.cloudwatch] Polling resource DomainName: my-ES-domainName
[2017-09-27T09:01:59,587][INFO ][logstash.inputs.cloudwatch] [AWS CloudWatch 200 0.05 0 retries] get_metric_statistics(:dimensions=>[{:name=>"DomainName",:value=>"my-ES-domainName"}],:end_time=>"2017-09-27T09:01:59+00:00",:metric_name=>"CPUUtilization",:namespace=>"AWS/ES",:period=>300,:start_time=>"2017-09-27T08:46:59+00:00",:statistics=>["SampleCount","Average","Minimum","Maximum","Sum"])

[2017-09-27T09:01:59,588][INFO ][logstash.inputs.cloudwatch] Polling resource ClientId: my-AWS-AccountId
[2017-09-27T09:01:59,647][INFO ][logstash.inputs.cloudwatch] [AWS CloudWatch 200 0.059 0 retries] get_metric_statistics(:dimensions=>[{:name=>"ClientId",:value=>"my-AWS-AccountId"}],:end_time=>"2017-09-27T09:01:59+00:00",:metric_name=>"CPUUtilization",:namespace=>"AWS/ES",:period=>300,:start_time=>"2017-09-27T08:46:59+00:00",:statistics=>["SampleCount","Average","Minimum","Maximum","Sum"])

Using AWS CLI:

aws cloudwatch get-metric-statistics --namespace AWS/ES --metric-name CPUUtilization --dimensions Name=DomainName,Value=my-ES-domainName Name=ClientId,Value=my-AWS-AccountId --start-time 2017-09-27T08:26:58.000Z --end-time 2017-09-27T08:41:58.000Z --period 300 --statistics SampleCount Average

Result:

{
    "Datapoints": [
        {
            "SampleCount": 5.0,
            "Timestamp": "2017-09-27T08:26:00Z",
            "Average": 8.0,
            "Unit": "Percent"
        },
        {
            "SampleCount": 5.0,
            "Timestamp": "2017-09-27T08:36:00Z",
            "Average": 6.0,
            "Unit": "Percent"
        },
        {
            "SampleCount": 5.0,
            "Timestamp": "2017-09-27T08:31:00Z",
            "Average": 7.6,
            "Unit": "Percent"
        }
    ],
    "Label": "CPUUtilization"
}

[tankhuu]

I found my problem, missing the option "combined". By adding it, I can collect the metric data now:

input {
  cloudwatch {
    namespace => "AWS/ES"
    metrics => [ "CPUUtilization" ]
    combined => true
    filters => {
      "DomainName" => "my-ES-domainName"
      "ClientId" => "my-AWS-AccountId"
    }
    aws_credentials_file => "/etc/logstash/aws_credential.yml"
    region => "ap-southeast-1"
  }
}

output {
  if [namespace] == "AWS/ES" {
    stdout {codec => rubydebug}
  }
}