aws-for-fluent-bit errors when disabling firehose, kinesis, and elastisearch plugins

[jallen-frb]

Describe the bug When using aws-for-fluent-bit helm chart, I disable everything but the cloudwatch plugin. This makes the container dump out errors for the other services.

  aws-for-fluent-bit:
    enabled: true
    source:
      repoURL: https://aws.github.io/eks-charts
      targetRevision: 0.1.16
      chart: aws-for-fluent-bit
      helm:
        cloudwatch:
          enabled: true
          region: us-east-1
        firehose:
          enabled: false
        kinesis:
          enabled: false
        elasticsearch:
          enabled: false
        tolerations:
          - key: "node-role.kubernetes.io/master"
            operator: "Exists"
            effect: "NoSchedule"
          - operator: "Exists"
            effect: "NoExecute"
          - operator: "Exists"
            effect: "NoSchedule"
        serviceAccount:
          create: true
          name: aws-for-fluent-bit-sa
          annotations:
            eks.amazonaws.com/role-arn: <my-role>

logs I get (redacted):

time="2022-05-11T12:20:07Z" level=error msg="[kinesis 0] PutRecords failed with AccessDeniedException: User: <my-role> is not authorized to perform: kinesis:PutRecords on resource: <a-kinesis-stream> because no identity-based policy allows the kinesis:PutRecords action\n\tstatus code: 400, request id: <request-id>\n"
time="2022-05-11T12:20:07Z" level=error msg="[kinesis 0] AccessDeniedException: User: is not authorized to perform: kinesis:PutRecords on resource:  because no identity-based policy allows the kinesis:PutRecords action\n\tstatus code: 400, request id: \n"
[2022/05/11 12:20:07] [ warn] [engine] chunk '1-1652271597.188086791.flb' cannot be retried: task_id=22, input=tail.0 > output=firehose.1
[2022/05/11 12:20:07] [ warn] [engine] chunk '1-1652271597.188086791.flb' cannot be retried: task_id=22, input=tail.0 > output=kinesis.2
[2022/05/11 12:20:07] [ warn] [engine] failed to flush chunk '1-1652271597.188086791.flb', retry in 10 seconds: task_id=22, input=tail.0 > output=es.3 (out_id=3)
[2022/05/11 12:20:09] [ warn] [engine] chunk '1-1652271281.422029229.flb' cannot be retried: task_id=21, input=tail.0 > output=es.3
[2022/05/11 12:20:09] [ info] [input] tail.0 resume (storage buf overlimit 22/128)
[2022/05/11 12:20:09] [ warn] [input] tail.0 paused (mem buf overlimit)
time="2022-05-11T12:20:11Z" level=error msg="[firehose 0] PutRecordBatch failed with AccessDeniedException: User:  is not authorized to perform: firehose:PutRecordBatch on resource:  because no identity-based policy allows the firehose:PutRecordBatch action\n\tstatus code: 400, request id: "
time="2022-05-11T12:20:11Z" level=error msg="[firehose 0] AccessDeniedException: User:  is not authorized to perform: firehose:PutRecordBatch on resource:  because no identity-based policy allows the firehose:PutRecordBatch action\n\tstatus code: 400, request id: \n"

Steps to reproduce Disable the firehose, kinesis, and elastisearch plugins and look at pod logs.

Expected outcome The firehose, kinesis, and elastisearch plugins are disabled with no pod log errors

Environment

• Chart name: aws-for-fluent-bit
• Chart version: 0.1.16
• Kubernetes version: 1.21
• Using EKS (yes/no), if so version? Yes 1.21

Additional Context:

[alt-dima]

interesting. Do you see those errors every second or just when fluentbit starts? Fluentbit do not work at all?

This is my configuration and it works fine (helm 0.1.17, eks 1.22)

cloudWatch:
  enabled: false

firehose:
  enabled: false

kinesis:
  enabled: false

elasticsearch:
  enabled: false

additionalOutputs: |
  [OUTPUT]
      Name                kafka

[johncmerfeld]

In my case the above fixed it. I hadn't explicitly disabled elasticsearch