MSK IAM auth without rotating short term keys/tokens?

[tooptoop4]

currently kowl works with: export KAFKA_SASL_AWSMSKIAM_SESSIONTOKEN='...redact' export KAFKA_SASL_AWSMSKIAM_SECRETKEY='..redact' export KAFKA_SASL_AWSMSKIAM_ACCESSKEY='..redact'

does it work with IAM role (like for java clients) instead of supplying aws keys/tokens that need to be rotated?

[weeco]

@tooptoop4 I'm unsure what you mean, the default is an IAM role?

[tooptoop4]

if I don't set KAFKA_SASL_AWSMSKIAM_SESSIONTOKEN and the other 2 variables I get sasl error

[twmb]

Access key and secret key are required, that's effectively your use and pass. Session token is not.

[tooptoop4]

but for java clients the access/secret key are not required

[twmb]

AFAIK, Java clients still load the access key / secret key, they just do this automatically by checking relevant environment variables or checking your ~/.aws/credentials automatically. The aws Go library can do this as well -- kowl just does not use that currently to auto-detect credentials.

When using a Java client, do you have any credentials at ~/.aws/credentials?

[tooptoop4]

no ~/.aws/credentials or env variables

[twmb]

Do you have any of the credentials located anywhere in options 1 through 7 as described in https://github.com/aws/aws-msk-iam-auth#configuring-a-kafka-client-to-use-aws-iam ?

[tooptoop4]

just 7 (http://169.254.169.254/latest/meta-data/iam/security-credentials/ returns a role) but I don't have to call that endpoint for the java client to work

[twmb]

Ok, so this is specific to the AWS library doing a lot of magic behind the scenes to try every possible way to load keys. The aws-sdk-go package provides some ways to automagically load creds, if @weeco wants to add them to kowl.

[jpandey55]

@weeco is anyone working on adding this feature?

[s2504s]

Hello, @weeco I read the code and saw the console binary uses AccessKey and SecretKey for AWS authentication. In my opinion, a lot of companies have not used AccessKey and SecretKey for several years. Currently, AWS SDK supports Role Based Auth or WebIdentity Token auth which are secure and preferred for AWS users. Also, AWS SDK supports AWS Default Credentials Provider Chain, and the developer can not worry about which Provider is used by a user. Let's shift the care of choosing authentication providers to a user :-)

[tooptoop4]

anyone got this working? I wonder if https://github.com/dpkp/kafka-python/pull/2255#issuecomment-1112236156 works

[m1mohamad]

I'm using AWS_MSK_IAM and IRSA that provide web identity and role arn is that supported ?

I get the following failure

{"level":"debug","ts":"2022-12-05T15:23:19.530Z","msg":"wrote SASLAuthenticate v2","source":"kafka_client","broker":"seed 0","by
tes_written":510,"write_wait":0.000008576,"time_to_write":0.000017558,"err":null}
{"level":"debug","ts":"2022-12-05T15:23:19.831Z","msg":"read SASLAuthenticate v2","source":"kafka_client","broker":"seed 0","byt
es_read":76,"read_wait":0.000021874,"time_to_read":0.301075236,"err":null}
{"level":"error","ts":"2022-12-05T15:23:19.831Z","msg":"unable to initialize sasl","source":"kafka_client","broker":"seed 0","er
r":"[ddeac038-a9b8-49b4-8461-a1738aed4108]: Internal error: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"debug","ts":"2022-12-05T15:23:19.831Z","msg":"connection initialization failed","source":"kafka_client","addr":"b-1.se
rvices-stage.kafka.eu-west-1.amazonaws.com:9098","broker":"seed 0","err":"[88888
8]: Internal error: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}
{"level":"debug","ts":"2022-12-05T15:23:19.831Z","msg":"kafka broker disconnected","source":"kafka_client_hooks","host":"b-1.ser
vices-stage.kafka.eu-west-1.amazonaws.com"}
{"level":"warn","ts":"2022-12-05T15:23:19.831Z","msg":"Failed to test Kafka connection, going to retry in 16s","remaining_retrie
s":1}

Using

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redpanda
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: infra
  source:
    repoURL: https://gitlab.com/infra.git
    targetRevision: master
    path: charts/redpanda
    helm:
      parameters: []
      releaseName: redpanda
      valueFiles: []
      values: |
        global:
          clusterName: stage-v1
          externalSecrets:
            enabled: true
        secret:
          create: true
        env:
          AWS_REGION: eu-west-1
          AWS_DEFAULT_REGION: eu-west-1
          POLLER_INTERVAL_MILLISECONDS: 300000
        serviceAccount:
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxx:role/redpanda-ui-stage
        resources:
          requests:
            cpu: 200m
            memory: 100Mi
          limits:
            cpu: 1
            memory: 200Mi
        # ingress:
        #   enabled: false
        #   annotations:
        #      kubernetes.io/ingress.class: platform
        #   hosts:
        #     - host: redpanda.stage.aaa.com
        #   tls:
        #     - secretName: root-wildcard-tls
        #       hosts:
        #         - "redpanda.stage.aaa.com"
        console:
          config:
            logger: 
              level: debug
            kafka:
              brokers:
                - b-1.services-stage.kafka.eu-west-1.amazonaws.com:9098
                - b-2.services-stage.kafka.eu-west-1.amazonaws.com:9098
              clientId: redpanda-console
              sasl:
                enabled: true
                mechanism: AWS_MSK_IAM
                username: developer
              tls:
                enabled: true

  destination:
    server: https://kubernetes.default.svc
    namespace: platform-services
  ignoreDifferences: []
  syncPolicy: {}

[jaredtbates]

This is preventing us from using Console in our environment, otherwise it looks super handy. Basically, we use IRSA to assign roles to K8s service accounts, and currently the accessKey and secretKey are hard-coded as the only AWS_MSK_IAM way to authenticate. Could you fall back to the default AWS SDK functionality if the access key and secret key aren't provided?

[weeco]

Can somebody lay down what is required to help you guys? Im not very familiar with AWS, but possibly open to pull request after a quick discussion on how to do it.

[jaredtbates]

@weeco thanks for the quick response! From what I understand after some quick looking, the library in use already has support for reading from the environment, as outlined at https://github.com/twmb/franz-go/tree/master/examples/sasl/aws_msk_iam. I'm pretty sure that as long as Console doesn't require the access key, secret, and session token, it will look them up in the environment using the AWS SDK, and will pick them up automatically.

[weeco]

@jaredtbates Thank you. By looking at the code I dont think it requires you to set the access key, see: https://github.com/redpanda-data/console/blob/0ba44b236b6ddd7191da015f44a9302fc13665ec/backend/pkg/config/kafka_sasl_aws_iam.go#L41-L44

What error do you get if you set the SASL mechanism to use MSK IAM but not specify any of these properties?

[contre95]

Hey sorry to jump in. But I'm in a similar situation with Console from Redpanda and I get the following error:

{"level":"error","ts":"2023-07-03T12:30:33.687Z","msg":"unable to initialize sasl","source":"kafka_client","broker":"seed 1","err":"[<redacted>]: Internal error: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}

[mislavmandaricaxilis]

Hi.

We're using Redpanda Console as a k8s deployment on EKS deployed using the official Helm chart, with IRSA authentication. Everything works well after initial deployment, but after exactly 1 hour, we start getting SASL_AUTHENTICATION_FAILED errors.

By default, the IRSA token has duration of 1h and I guess the problem is that there is no token refresh implemented. Looking at the code here, it looks like that is indeed the case - token is fetched at startup and reused for all subsequent requests. That works until the token expires, after which all requests fail as unauthenticated.

Here is the config we use for Redpanda Console and for the service account:

console:
  config:
    kafka:
      brokers:
        - b-1.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-2.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-3.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
      sasl:
        enabled: true
        mechanism: AWS_MSK_IAM
      tls:
        enabled: true

Here is the config we use for service account:

serviceAccount:
  create: true
  automountServiceAccountToken: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/something

And here is the error log after one hour, once the token expires:

{"level":"error","ts":"2024-05-02T08:43:12.242Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"2","err":"[55396e20-7c33-4065-b9bd-fb7413070759]: Access denied: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}

We could increase token expiry duration, but that would just reduce the frequency of the problem, as we can't make it non-expire. it would be better to change the code to not persist and reuse the same token. I'm willing to make a PR with this change, if you confirm that the assessment is correct and suggested improvement is the way to go.

[ialejandro]

Hi.

We're using Redpanda Console as a k8s deployment on EKS deployed using the official Helm chart, with IRSA authentication. Everything works well after initial deployment, but after exactly 1 hour, we start getting SASL_AUTHENTICATION_FAILED errors.

By default, the IRSA token has duration of 1h and I guess the problem is that there is no token refresh implemented. Looking at the code here, it looks like that is indeed the case - token is fetched at startup and reused for all subsequent requests. That works until the token expires, after which all requests fail as unauthenticated.

Here is the config we use for Redpanda Console and for the service account:

console:
  config:
    kafka:
      brokers:
        - b-1.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-2.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
        - b-3.xxx.yyyyyy.zz.kafka.eu-central-1.amazonaws.com:9098
      sasl:
        enabled: true
        mechanism: AWS_MSK_IAM
      tls:
        enabled: true

Here is the config we use for service account:

serviceAccount:
  create: true
  automountServiceAccountToken: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/something

And here is the error log after one hour, once the token expires:

{"level":"error","ts":"2024-05-02T08:43:12.242Z","logger":"kafka_client","msg":"unable to initialize sasl","broker":"2","err":"[55396e20-7c33-4065-b9bd-fb7413070759]: Access denied: SASL_AUTHENTICATION_FAILED: SASL Authentication failed."}

We could increase token expiry duration, but that would just reduce the frequency of the problem, as we can't make it non-expire. it would be better to change the code to not persist and reuse the same token. I'm willing to make a PR with this change, if you confirm that the assessment is correct and suggested improvement is the way to go.

Same case here. Did you find any solution?

[hongbo-miao]

I can confirm https://github.com/redpanda-data/console/pull/1253 resolved the issue, thanks @mislavmandaricaxilis! ☺️

Here is my custom Helm chart values.yaml:

image:
  registry: docker.redpanda.com
  repository: redpandadata/console
  # https://hub.docker.com/r/redpandadata/console/tags
  tag: v2.6.1
console:
  config:
    kafka:
      brokers:
        - b-1.hmkafka.xxxxxx.xx.kafka.us-west-2.amazonaws.com:9098
        - b-2.hmkafka.xxxxxx.xx.kafka.us-west-2.amazonaws.com:9098
        - b-3.hmkafka.xxxxxx.xx.kafka.us-west-2.amazonaws.com:9098
      sasl:
        enabled: true
        mechanism: AWS_MSK_IAM
      tls:
        enabled: true
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/RedpandaConsoleRole-hm-redpanda-console

Note the latest Helm chart 0.7.26 is still using an old Redpanda Console v2.4.6. You need set tag: v2.6.1.