syedriko
commented
3 years ago On a second thought, I'm not sure this is an AWS SDK issue necessarily. Consider this test snippet, which is supposed to fail a create_log_group() call at different log levels:
require 'aws-sdk-cloudwatchlogs'
require 'time'
require 'logger'
logger = Logger.new(STDOUT)
logger.level = 'UNKNOWN'
cwl_client = Aws::CloudWatchLogs::Client.new(
region: 'us-east-2',
access_key_id: <...>,
secret_access_key: <...>,
logger: logger
)
resp = cwl_client.delete_log_group(
log_group_name: 'foo'
)
resp = cwl_client.create_log_group(
log_group_name: 'foo'
)
if !resp.successful?
puts 'create_log_group failed' + resp
exit(1)
end
['DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL', 'UNKNOWN'].each do |level|
begin
puts "logger.level: " + level
logger.level = level
resp = cwl_client.create_log_group(
log_group_name: 'foo'
)
rescue
end
endThe output this produces is
$ ruby ./test.rb
logger.level: DEBUG
D, [2021-08-11T23:21:09.354722 #28770] DEBUG -- : [Aws::CloudWatchLogs::Client 400 0.054844 0 retries] create_log_group(log_group_name:"foo") Aws::CloudWatchLogs::Errors::ResourceAlreadyExistsException The specified log group already exists
logger.level: INFO
logger.level: WARN
logger.level: ERROR
logger.level: FATAL
logger.level: UNKNOWN
$ that is, AWS API calls are only logged at DEBUG level.
Problem
When invoked from the fluent-plugin-cloudwatch-logs plugin, the AWS SDK logs every AWS API call it makes, no matter what global logging level is set for fluentd. The end result is that fluentd log grows proportionately to the "payload" log traffic that is forwarded to CloudWatch.
I believe the root cause is here: https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/lib/aws-sdk-core/plugins/logging.rb#L53, because this line
always logs at the current log level.
However, an issue in this repo provides the necessary application-level context for what is arguably a defect in the AWS SDK.
Steps to replicate
Example fluentd config
fluentd.conf:
Fluentd logs
In response to
the following is seen in fluentd logs. Note that the log messages from Aws::CloudWatchLogs::Client are issued at every log level:
Expected Behavior
I would expect that AWS API calls are logged at a certain log level (debug) and above, but not below. For example, with the debug log level hardcoded in the AWS SDK code line mentioned above:
Fluentd and CloudWatchLogs plugin versions
fluentd --versionortd-agent --versionfluent-gem list,td-agent-gem listor your Gemfile.lockLOCAL GEMS
aws-eventstream (1.1.0) aws-partitions (1.396.0) aws-sdk-cloudwatchlogs (1.38.0) aws-sdk-core (3.109.3) aws-sigv4 (1.2.2) bigdecimal (default: 1.3.4) cmath (default: 1.0.0) concurrent-ruby (1.1.9) cool.io (1.7.1) csv (default: 1.0.0) date (default: 1.0.0) did_you_mean (1.2.0) dig_rb (1.0.1) etc (default: 1.0.0) fcntl (default: 1.0.0) fiddle (default: 1.0.0) fileutils (default: 1.0.2) fluent-plugin-cloudwatch-logs (0.7.6) fluentd (1.7.4) http_parser.rb (0.6.0) io-console (default: 0.4.6) ipaddr (default: 1.2.0) jmespath (1.4.0) json (default: 2.1.0) minitest (5.10.3) msgpack (1.4.2) net-telnet (0.1.1) openssl (default: 2.1.2) power_assert (1.1.1) psych (default: 3.0.2) rake (12.3.0) rdoc (default: 6.0.1) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) sigdump (0.2.4) stringio (default: 0.0.1) strptime (0.2.5) strscan (default: 1.0.0) test-unit (3.2.7) tzinfo (2.0.4) tzinfo-data (1.2021.1) webrick (default: 1.4.2) xmlrpc (0.3.0) yajl-ruby (1.4.1) zlib (default: 1.0.0) $