logstash-plugins / logstash-input-cloudwatch

A Logstash input to pull events from the Amazon Web Services CloudWatch API
Apache License 2.0
43 stars 28 forks source link

Decrease logging verbosity at INFO level #26

Closed lukewaite closed 6 years ago

lukewaite commented 7 years ago

The verbosity level of the logstash-input-cloudwatch plugin when running INFO level logging is quite high.

Currently, using this config I get the following info level messages on every iteration:

input {
    cloudwatch {
        namespace => "AWS/EC2"
        metrics => 'CPUUtilization'
        metrics => 'DiskReadOps'
        metrics => 'DiskWriteOps'
        metrics => 'NetworkIn'
        metrics => 'NetworkOut'
        metrics => 'CPUCreditBalance'
        metrics => 'CPUCreditUsage'
        metrics => 'StatusCheckFailed_Instance'
        metrics => 'StatusCheckFailed_System'
        filters => { "tag:Monitoring" => "Yes" }
        region => "us-east-1"
        interval => 600
    }
}

This means in an environment with 100 instances (I am currently running in the 90s on average) I get approximately 918 log messages per 10 minute iteration.

(start) + (metrics) + (metrics * instances) = (total log messages)
   1    +     18    +        9 * 100        = 919

Not only is this a lot of logs, but the logs are exceptionally verbose, as most include JSON payloads related to the message.

As compared to, for instance, the logstash-input-s3 plugin, where only info level log messages are in regards to registering the plugin (https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L67, https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L278-L282), this seems excessive. It certainly made things difficult for me in terms of troubleshooting an issue recently.