[bitnami/fluentd] Fluend-aggregator with AWS EKS IRSA gives `No valid AWS credentials found`

damvinod commented 1 year ago

Name and Version

bitnami/fluentd-5.0.18

What steps will reproduce the bug?

We have a use-case to use fluend as aggregator to push logs to Cloudwatch. We are enabling IRSA for the fluentd-aggregator pod so that it has necessary permission to pipe logs to Cloudwatch.

fluentd image is build with below config.

DockerFile

FROM bitnami/fluentd:1.13.3

COPY Gemfile /opt/bitnami/fluentd/
RUN bundle install

RUN fluent-gem install 'fluent-plugin-cloudwatch-logs' -v 0.14.0 \
 && fluent-gem install 'fluent-plugin-route' -v 1.0.0 \
 && fluent-gem install 'fluent-plugin-remote_syslog' -v 1.0.0 \
 && fluent-gem install 'fluent-plugin-aws-elasticsearch-service' -v 2.4.1

Gemfile

source "https://rubygems.org"

gem 'aws-sdk-cloudwatchlogs', '1.45.0'
gem 'aws-sdk-core', '3.121.1'
gem 'aws-partitions', '1.479.0'

Are you using any custom parameters or values?

No response

What is the expected behavior?

fluentd-aggregator pod should be able to pipe logs to Cloudwatch.

What do you see instead?

2023-01-10 05:47:40 +0000 [error]: #0 unexpected error error_class=RuntimeError error="No valid AWS credentials found."
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:110:in `block in credentials'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:114:in `inspect'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:71:in `inspect'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:71:in `redact'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:65:in `block in redacted_string'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:64:in `each'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:64:in `reduce'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:64:in `redacted_string'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/redacted.rb:49:in `inspect'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:331:in `rescue in perform_request'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/base.rb:278:in `perform_request'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-transport-7.13.3/lib/elasticsearch/transport/client.rb:192:in `perform_request'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-api-7.13.3/lib/elasticsearch/api/actions/info.rb:37:in `info'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:490:in `block in handle_last_seen_es_major_version'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:40:in `retry_operate'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:74:in `block in configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `each'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in `add_match'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:714:in `block in run_worker'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in `main_process'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in `run_worker'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in `<top (required)>'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/ruby/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/ruby/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:85:in `require'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/bin/fluentd:23:in `load'
  2023-01-10 05:47:40 +0000 [error]: #0 /opt/bitnami/fluentd/bin/fluentd:23:in `<main>'

Additional information

Verified the below:

IAM OIDC provider is configured correctly as it works for other pod
IAM role trust policy is updated correctly with pods namespace and service account
Service account of the pod is configured correctly to use the IAM role

javsalgar commented 1 year ago

Hi,

Looking at the issue it seems to me that the problem is more on the plugin itself than the Bitnami packaging of fluentd. Did you check with the plugin developers?

damvinod commented 1 year ago

Thanks @javsalgar.

The issue was resolved after configuring the IAM role as part of fluend configuration which is specific to the pluginfluent-plugin-aws-elasticsearch-service based on the below documentation.

https://github.com/atomita/fluent-plugin-aws-elasticsearch-service#eks

bitnami / charts