Closed lukewaite closed 6 years ago
The verbosity level of the logstash-input-cloudwatch plugin when running INFO level logging is quite high.
logstash-input-cloudwatch
Currently, using this config I get the following info level messages on every iteration:
input { cloudwatch { namespace => "AWS/EC2" metrics => 'CPUUtilization' metrics => 'DiskReadOps' metrics => 'DiskWriteOps' metrics => 'NetworkIn' metrics => 'NetworkOut' metrics => 'CPUCreditBalance' metrics => 'CPUCreditUsage' metrics => 'StatusCheckFailed_Instance' metrics => 'StatusCheckFailed_System' filters => { "tag:Monitoring" => "Yes" } region => "us-east-1" interval => 600 } }
This means in an environment with 100 instances (I am currently running in the 90s on average) I get approximately 918 log messages per 10 minute iteration.
(start) + (metrics) + (metrics * instances) = (total log messages) 1 + 18 + 9 * 100 = 919
Not only is this a lot of logs, but the logs are exceptionally verbose, as most include JSON payloads related to the message.
As compared to, for instance, the logstash-input-s3 plugin, where only info level log messages are in regards to registering the plugin (https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L67, https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L278-L282), this seems excessive. It certainly made things difficult for me in terms of troubleshooting an issue recently.
logstash-input-s3
The verbosity level of the
logstash-input-cloudwatch
plugin when running INFO level logging is quite high.Currently, using this config I get the following info level messages on every iteration:
This means in an environment with 100 instances (I am currently running in the 90s on average) I get approximately 918 log messages per 10 minute iteration.
Not only is this a lot of logs, but the logs are exceptionally verbose, as most include JSON payloads related to the message.
As compared to, for instance, the
logstash-input-s3
plugin, where only info level log messages are in regards to registering the plugin (https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L67, https://github.com/logstash-plugins/logstash-input-s3/blob/master/lib/logstash/inputs/s3.rb#L278-L282), this seems excessive. It certainly made things difficult for me in terms of troubleshooting an issue recently.