[Kafka] KeeperErrorCode = NoAuth for /config/users/user

[rdziwinski]

Name and Version

bitnami/kafka:24.0.14

What architecture are you using?

amd64

What steps will reproduce the bug?

env: k8s: 1.29.4 helm: 3.14.4 chart: 24.0.14

Hello, I need upgrade Kafka 3.5 from Zookeeper mode to Kraft. I perform upgrade helm chart from 23 to 24. After that kafka broker was restarted due error below.

Are you using any custom parameters or values?

Before: v23

auth: 
  clientProtocol: mtls
  externalClientProtocol: ""
  interBrokerProtocol: sasl
  sasl:
    mechanisms: plain,scram-sha-256,scram-sha-512
    interBrokerMechanism: plain 
    jaas:    
      interBrokerUser: kafka_broker
      interBrokerPassword:
      zookeeperUser: kafus
      zookeeperPassword:
      existingSecret: passwords

  tls:
    type: pem
    pemChainIncluded: false
    existingSecrets:
      - kafka-0-tls
      - kafka-1-tls
      - kafka-2-tls
    autoGenerated: false

allowPlaintextListener: false
interBrokerListenerName: INTERNAL
authorizerClassName: "kafka.security.authorizer.AclAuthorizer"

After: v24

extraConfig: |
  auto.create.topics.enable = false
  num.partitions = 3
  default.replication.factor = 3
  delete.topic.enable = true
  log.flush.interval.ms = 1000
  log.retention.hours = 60
  log.retention.bytes = 1073741824
  log.segment.bytes = 1073741824
  allow.everyone.if.no.acl.found = true
  super.users=User:admin
  authorizer.class.name = org.apache.kafka.metadata.authorizer.StandardAuthorizer
  zookeeper.set.acl=false

listeners:
  client:
    containerPort: 9092
    protocol: SSL
    name: CLIENT
    sslClientAuth: required
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: SASL_PLAINTEXT
  interbroker:
    containerPort: 9094
    protocol: SASL_PLAINTEXT
    name: INTERNAL
  external:
    containerPort: 9095
    protocol: SSL
    name: EXTERNAL
  sslClientAuth: required

sasl:
  enabledMechanisms: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
  interBrokerMechanism: PLAIN
  controllerMechanism: PLAIN
  zookeeper:
    user: kafus
    password: "password"
  interbroker:
    user: kafka_broker
    password: "password"
  controller:
    user: controller_user
    password: "password"
  client:
    users:
      - user
    passwords: "password"

tls:
  type: PEM
  pemChainIncluded: false
  existingSecret: kafka-all-tls
  autoGenerated: false
  endpointIdentificationAlgorithm: https
  sslClientAuth: required

controller:
  replicaCount: 1
  controllerOnly: true
  zookeeperMigrationMode: true

broker:
  zookeeperMigrationMode: false
  replicaCount: 3
  minId: 0

kraft:
  enabled: true
  clusterId: "ididididid"

zookeeper:
  clusterDomain: k8s.infra.k.pl
  enabled: true
  replicaCount: 3
  auth:
    client:
      enabled: true
      clientUser: "kafus"
      clientPassword: "password"
      serverUsers: "kafus"
      serverPasswords: "password"

What do you see instead?

Kafka broker restarted with error below:

kafka 10:21:53.13 DEBUG ==> Creating user user in zookeeper
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Error while executing config command with args '--zookeeper kafka-zookeeper:2181 --alter --add-config SCRAM-SHA-256=[iterations=8192,password=bitnami],SCRAM-SHA-512=[password=bitnami] --entity-type users --entity-name user'
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /config/users/user
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
    at kafka.zookeeper.AsyncResponse.resultException(ZooKeeperClient.scala:563)
    at kafka.zk.KafkaZkClient.getEntityConfigs(KafkaZkClient.scala:435)
    at kafka.zk.AdminZkClient.fetchEntityConfig(AdminZkClient.scala:552)
    at kafka.admin.ConfigCommand$.alterConfigWithZk(ConfigCommand.scala:160)
    at kafka.admin.ConfigCommand$.processCommandWithZk(ConfigCommand.scala:121)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:95)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

Additional information

I'm confused that in the new server.properties is JKS certs, instead of PEM as i declaration in tls.type=PEM

# TLS configuration
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.keystore.location=/opt/bitnami/kafka/config/certs/kafka.keystore.jks
ssl.truststore.location=/opt/bitnami/kafka/config/certs/kafka.truststore.jks
ssl.keystore.password=keystore-password
ssl.truststore.password=truststore-password

I'm trying to add acl for allow all for user "user" in zookeeper but after login in zookeeper pods and user jaas.conf with auth's I don't have permission.

[carrodher]

There are some hints about upgrading to that version at https://github.com/bitnami/charts/tree/main/bitnami/kafka#to-2400, did you take a look?

[rdziwinski]

Yes, maybe i make some mistake and don't see it :( But yes, i read it and change needed values.

[carrodher]

The issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being upgraded in your specific environment, or tied to a specific scenario that is not easy to reproduce on our side.

If you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[rdziwinski]

Do you have any ideas why despite set tls.type=PEM in server.properties is still ssl.keystore.type=JKS and another settings related with JKS as i show in first post?

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.