[fluentd-elasticsearch] helm chart does not support authentication using AWS IAM and Kubernetes service account

[kartik-moolya]

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Version of Helm and Kubernetes: HELM3 Kubernetes: 1.15 hosted on AWS EKS

Which chart in which version: fluentd-elasticsearch, 6.1.1

What happened: I used kiam to authenticate my elasticsearch hosted on AWS, it worked fine. I'm now trying to use AWS IAM with Kubernetes Service Account but it fails with error below. Is there a way we can verify if it uses the supported AWS SDK version as per. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

2020-06-23 14:44:05 +0000 [error]: unexpected error error_class=NoMethodError error="undefined method `[]' for nil:NilClass"
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluent-plugin-elasticsearch-3.7.1/lib/fluent/plugin/out_elasticsearch.rb:395:in `detect_es_major_version'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluent-plugin-elasticsearch-3.7.1/lib/fluent/plugin/out_elasticsearch.rb:264:in `block in configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluent-plugin-elasticsearch-3.7.1/lib/fluent/plugin/elasticsearch_index_template.rb:35:in `retry_operate'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluent-plugin-elasticsearch-3.7.1/lib/fluent/plugin/out_elasticsearch.rb:263:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/plugin.rb:164:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/agent.rb:130:in `add_match'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/agent.rb:72:in `block in configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/agent.rb:64:in `each'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/agent.rb:64:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/label.rb:31:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/root_agent.rb:147:in `block in configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/root_agent.rb:147:in `each'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/root_agent.rb:147:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/engine.rb:131:in `configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/engine.rb:96:in `run_configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/supervisor.rb:804:in `run_configure'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/supervisor.rb:550:in `block in run_worker'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/supervisor.rb:733:in `main_process'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/supervisor.rb:546:in `run_worker'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/lib/fluent/command/fluentd.rb:320:in `<top (required)>'
  2020-06-23 14:44:05 +0000 [error]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2020-06-23 14:44:05 +0000 [error]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2020-06-23 14:44:05 +0000 [error]: /var/lib/gems/2.5.0/gems/fluentd-1.6.3/bin/fluentd:8:in `<top (required)>'
  2020-06-23 14:44:05 +0000 [error]: /usr/local/bin/fluentd:23:in `load'
  2020-06-23 14:44:05 +0000 [error]: /usr/local/bin/fluentd:23:in `<main>'

What you expected to happen: Should be able to authencticate to AWS Elasticsearch

How to reproduce it (as minimally and precisely as possible): Just try implementing the AWS IAM to service account feature

Anything else we need to know: Values.yaml

awsSigningSidecar:
  enabled: true
podAnnotations:
elasticsearch:
  auth:
    enabled: false
    user: ""
    password: ""
  bufferChunkLimit: "2M"
  bufferQueueLimit: 8
  host: "vpc-xxxxg-store-xxxxwest-1.es.amazonaws.com"
  port: 443
  logstashPrefix: "ixxxgs"
  scheme: "https"
  sslVerify: true
  sslVersion: "TLSv1_2"
  logLevel: "info"

# RBAC is not working, fluentd still creates a services account using the template fullname
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::${aws_account_id}:role/${es_role_name}"

resources:
  limits:
    cpu: 300m
    memory: 2Gi
  requests:
    cpu: 100m
    memory: 1Gi

serviceMonitor:
  enabled: true
  labels:
    release: prom

extraConfigMaps:
  exclude-calico-node.conf: |-
    # Exclude INFO and TRACE level log entries of calico-node and
    # calico-typha.
    # Sample log entry:
    # 2020-01-09 11:15:31.613 [INFO][6] health.go …
    <filter kubernetes.**>
      @type grep
      <and>
        <exclude>
          key message
          pattern /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} (\[INFO\]|\[TRACE\])/
        </exclude>
        <exclude>
          key $.kubernetes.container_name
          pattern /^(calico-node|calico-typha)$/
        </exclude>
      </and>
    </filter>
  exclude-cloudwatch-agent.conf: |-
    # Exclude I! (info) and D! (debug) level log entries of
    # cloudwatch-agent.
    # Sample log entry:
    # 2020-01-09T02:32:33Z I! Loaded outputs: cloudwatchlogs
    <filter kubernetes.**>
      @type grep
      <and>
        <exclude>
          key message
          pattern /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z (I!|D!)/
        </exclude>
        <exclude>
          key $.kubernetes.container_name
          pattern /^cloudwatch-agent$/
        </exclude>
      </and>
    </filter>

tolerations:
- operator: "Exists"```

[kartik-moolya]

I'm using below aws-es-proxy image abutaha/aws-es-proxy:0.9

I have also tried using the latest image : v1.1 it fails with Received 403 from AWSAuth, invalidating credentials for......

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kartik-moolya]

Is anyone looking into this ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.