[bitnami/kafka] Required configuration for Zookeeper mode following Kafka Chart Refactor in 24.0.0

PapaAAnthony commented 1 year ago

Name and Version

bitnami/kafka 25.1.12

What architecture are you using?

None

What steps will reproduce the bug?

Running a 3 node Kafka cluster with KRaft disabled and Zookeeper enabled and listeners set to plaintext.

kraft:
enabled: false
zookeeper:
enabled: true
listeners:
client:
  protocol: PLAINTEXT
controller:
  protocol: PLAINTEXT
interbroker:
  protocol: PLAINTEXT
external:
  protocol: PLAINTEXT
broker:
replicaCount: 3

What is the expected behavior?

I would expect the brokers to start with KRaft disabled, as well as other default setting that allow the cluster to start up.

What do you see instead?

I am seeing the below error:

kafka 09:26:30.11                                                                                                                                                                                                                                     
kafka 09:26:30.11 Welcome to the Bitnami kafka container                                                                                                                                                                     
kafka 09:26:30.11 Subscribe to project updates by watching https://github.com/bitnami/containers                                                      
kafka 09:26:30.11 Submit issues and feature requests at https://github.com/bitnami/containers/issues    
kafka 09:26:30.11                                                                                                                                                                                                                                           kafka 09:26:30.11 INFO  ==> ** Starting Kafka setup **                                                                                                                                                                                                     
kafka 09:26:30.16 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=true. For safety reasons, do not use this flag in a production environment.                                                                                          
kafka 09:26:30.16 INFO  ==> Initializing Kafka...                                                                                                                                                                                                          
kafka 09:26:30.17 INFO  ==> No injected configuration files found, creating default config files   
kafka 09:26:30.25 INFO  ==> Initializing KRaft...                                                                                                                                                                                                          
kafka 09:26:30.25 WARN  ==> KAFKA_KRAFT_CLUSTER_ID not set - If using multiple nodes then you must use the same Cluster ID for each one                                                                                                                    
kafka 09:26:31.39 INFO  ==> Generated Kafka cluster ID ' '                                                                                                                                                                            
kafka 09:26:31.39 INFO  ==> Formatting storage directories to add metadata...                                                                                                                                                                              
kafka 09:26:32.82 INFO  ==> ** Kafka setup finished! **                                                                                                                                                                                                                           kafka 09:26:32.84 INFO  ==> ** Starting Kafka **                                                                                                                                                                               
2023-10-03 09:26:33,570] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)                                                                                                                                
[2023-10-03 09:26:33,935] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)                                                                             
[2023-10-03 09:26:34,021] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)                                                                                                                                                                        
kafka.common.InconsistentNodeIdException: Configured node.id `1` doesn't match stored node.id `0' in meta.properties. If you moved your data, make sure your configured controller.id matches. If you intend to create a new broker, you should remove all 
 data in your data directories (log.dirs).                                                                                                                                                                                                            
│     at kafka.server.KafkaRaftServer$.initializeLogDirs(KafkaRaftServer.scala:163)                                                                                                                                                                          │
│     at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:57)                                                                                                                                                                                       │
│     at kafka.Kafka$.buildServer(Kafka.scala:83)                                                                                                                                                                                                            │
│     at kafka.Kafka$.main(Kafka.scala:91)                                                                                                                                                                                                                   │
│     at kafka.Kafka.main(Kafka.scala)

Additional information

I would've thought that setting kraft.enabled to false would have set KAFKA_ENABLE_KRAFT preventing the brokers from initializing KRaft.

So I'm just wondering if there is any documentation or examples that will allow me to run with the settings I had previously in v23.0.0 which were mostly the defaults that came in the chart.

migruiz4 commented 1 year ago

Hi @PapaAAnthony,

I haven't been able to reproduce your issue on a fresh install.

Using the values.yaml you provided, the following error would show:

$ helm install kafka bitnami/kafka -f values.yaml 
Error: INSTALLATION FAILED: execution error at (kafka/templates/NOTES.txt:333:4): 
VALUES VALIDATION:

kafka: Zookeeper mode - Controller nodes not supported
    Controller replicas have been enabled in Zookeeper mode, set controller.replicaCount to zero or enable migration mode to migrate to Kraft mode

After adding controller.replicaCount: 0, Kafka successfully deployed in Zookeeper mode.

In your case, did you have any PVC from previous deployment or are you performing an upgrade from a previous version?

migruiz4 commented 1 year ago

Please notice that the bitnami/kafka image also received changes in the major version 24.x.x. If you are using a custom image or an older tag, please make sure to update it.

PapaAAnthony commented 1 year ago

Hi @migruiz4,

Apologies I missed off the below config:

controller:
   replicaCount: 0

Additionally I should add that I was upgrading from an established cluster that already contained data in the PVC so I followed the steps outlined here: https://github.com/bitnami/charts/tree/main/bitnami/kafka#to-2400

migruiz4 commented 1 year ago

Hi @PapaAAnthony,

In that case, could you please provide more details about your upgrade? What version were you upgrading from? What values were you using in that previous version?

The error message aims that server.properties configures node.id: 1 while meta.properties (PVC) has configured node.id: 0.

The chart logic considers this case and should use the node.id stored at meta.properties:

https://github.com/bitnami/charts/blob/3cb44e376a472ca6721866b09f6d0ab412338cbc/bitnami/kafka/templates/scripts-configmap.yaml#L336-L357

migruiz4 commented 1 year ago

Using your values.yaml, rendered logic would be:

    if [[ -f "/bitnami/kafka/data/meta.properties" ]]; then
        if grep -q "broker.id" /bitnami/kafka/data/meta.properties; then
          ID="$(grep "broker.id" /bitnami/kafka/data/meta.properties | awk -F '=' '{print $2}')"
          kafka_conf_set "$KAFKA_CONFIG_FILE" "node.id" "$ID"
        else
          ID="$(grep "node.id" /bitnami/kafka/data/meta.properties | awk -F '=' '{print $2}')"
          kafka_conf_set "$KAFKA_CONFIG_FILE" "node.id" "$ID"
        fi
    else
        ID=$((POD_ID + KAFKA_MIN_ID))
        kafka_conf_set "$KAFKA_CONFIG_FILE" "broker.id" "$ID"
    fi

It would be helpful if you could also share the contents of /opt/bitnami/kafka/config/server.properties and /bitnami/kafka/data/meta.properties inside your pods.

PapaAAnthony commented 1 year ago

So I was upgrading from 23.0.7, in this case the value overrides I was using were minimal:

replicaCount: 3
kraft:
  enabled: false
zookeeper:
  enabled: true

I managed to get past the above error by adding the following:

extraEnvVars:
    - name: KAFKA_ENABLE_KRAFT
      value: "false"

My main question is why I now have to explicitly add the environment variable where previously it was part of the chart? Further to that I want to understand what I need to now add to get the cluster to run as it was in 23.0.7 which I upgraded from.

migruiz4 commented 1 year ago

@PapaAAnthony the env variable KAFKA_ENABLE_KRAFT has been removed and the chart has been radically refactored.

It now does not rely on container logic, but the configuration is now generated as a Helm template and modified in an initContainer. This change was motivated to improve the security of the chart by adding the readOnlyRootFilesystem flag.

It is an important detail if you are using a custom value for the image.tag as the image also included major changes.

Additionally please share these details so we can further help you:

It would be helpful if you could also share the contents of /opt/bitnami/kafka/config/server.properties and /bitnami/kafka/data/meta.properties inside your pods.

PapaAAnthony commented 1 year ago

Hi, we are using version 3.5.1 of the bitnami/kafka image.

I can't currently access the server.properties/meta.properties as the pods are currently failing with the below:

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://:9092,CONTROLLER://:9093': No security protocol defined for listener PLAINTEXT

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

migruiz4 commented 1 year ago

Hi @PapaAAnthony,

I'm sorry for the late response.

That error message displays that your Kafka server.properties is configured with listeners=PLAINTEXT://:9092,CONTROLLER://:9093

That differs from the chart's default listeners, which are CLIENT,INTERNAL,CONTROLLER:

    listeners=CLIENT://:9092,INTERNAL://:9094,CONTROLLER://:9093
    advertised.listeners=CLIENT://advertised-address-placeholder:9092,INTERNAL://advertised-address-placeholder:9094
  listener.security.protocol.map=CLIENT:SASL_PLAINTEXT,INTERNAL:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT

Is it possible you have set listeners.overrideListeners in your values.yaml?

It would be helpful if you could also share the following:

Content of the kafka--configuration configmap: kubectl get cm kafka-controller-configuration -o yaml.
Pod description: kubectl describe pod kafka-<role>-0.
Any additional values.yaml you did not provide in the issue description.

github-actions[bot] commented 11 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 11 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

bitnami / charts