[fluent-bit] Unable to authenticate Opensearch with custom IAM role in EKS 1.24

[bogdandisc]

Environment:

EKS 1.23
fluent-bit chart version 0.30.4
fluent-bit app version 2.0.14

I am trying to deploy fluent-bit using the standard helm chart:

resource "helm_release" "fluent-bit" {
  repository       = "https://fluent.github.io/helm-charts"
  version          = var.versions.fluent_bit_chart
  chart            = "fluent-bit"
  name             = "fluent-bit"
  namespace        = "kube-system"
  create_namespace = false
  atomic           = true

  values = [
    templatefile("${path.module}/templates/fluent-bit-helm-values.yaml", {
      role_arn = "role_arn" 
      ...
    })
  ]
}

Have tried to create a new service account via the helm chart:

serviceAccount:
  create: true
  name: fluent-bit
  annotations:
    eks.amazonaws.com/role-arn: ${role_arn}

And also tried to create a new service account and attach it in the values.yaml file. In both cases I can see the service account getting attached to the fluent-bit pods and the trust policy working. There's no reason that the IRSA token is not used by the fluent-bit role.

I am also aware that in 1.24 secrets are not created automatically, so I am creating one in terraform and attaching it to the service account:

resource "kubernetes_secret" "fluent_bit" {
  metadata {
    generate_name = "fluent-bit-"
    namespace     = "kube-system"
    annotations = {
      "kubernetes.io/service-account.name" = kubernetes_service_account.fluent_bit_sa.metadata[0].name
    }
  }

  type                           = "kubernetes.io/service-account-token"
  wait_for_service_account_token = true
}

However the fluent-bit pods are not authenticating using the fluent-bit service account and are using the worker node IAM role instead:

23/06/14 09:54:29] [error] [output:opensearch:opensearch-application] HTTP status=403 URI=/_bulk, response:
{"Message":"User: arn:aws:sts::****:assumed-role/eks_role is not authorized to perform: es:ESHttpPost because no identity-based policy allows the es:ESHttpPost action"}

[RobertKarolczuk]

Hi @bogdandisc have You resolved the issue? Unfortunately I have similar issue and according to the AWS documentation You have to attach proper policy to the OS, and additionally You have to enable AWS Signature Version 4. \ Even if you configure a completely open resource-based access policy, all requests to the OpenSearch Service configuration API must be signed. If your policies specify IAM roles or users, requests to the OpenSearch APIs also must be signed using AWS Signature Version 4.

I found in Fluentbit Docuemtnation the parameter AWS_Auth - Enable AWS Sigv4 Authentication for Amazon OpenSearch Service

but I use helm chart and to be honest I don't know how to use this parameter in my case... Below section from values.yaml

clusterOutputs:
  - name: opensearch-output
    spec:
      opensearch:
        host: vpc-domain-name-xxxxxxxxxxxxx.us-east-1.es.amazonaws.com
        port: 443
        scheme: https
        logstash_format: true
        logstash_prefix: ${tag}
        index_name: ${tag}-%Y.%m.%d
        include_timestamp: true
        reconnect_on_error: true
        log_os_400_reason: true
        buffer:
          path: /buffers/opensearch
          type: file
          timekey: 1m
          timekey_wait: 30s
          timekey_use_utc: true

Do You have any suggestions/ideas?

Thanks

[skybalsamoan]

have you given necessary wite grants on opensearch side? you have to map your role with necessary permission https://opensearch.org/docs/latest/security/access-control/users-roles/ Creation of k8s secrets was not necessary from my experience In your output conf you have also to add AWS_Auth On AWS_Region eu-west-1