search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.83k stars 9.12k forks source link

[bitnami/kafka] Config and Modify Listeners as Desired. #28547

Open sinamohsenifar opened 1 month ago

sinamohsenifar commented 1 month ago

Name and Version

bitnami/kafka

What is the problem this feature will solve?

in some projects, the clients can't see Kubernetes cluster workers. in situations like this, we route their traffic through a firewall virtual IP or from another secure server. we need to configure and modify listeners freely with Kubernetes setup with Nodeport configs.

What is the feature you are proposing to solve the problem?

access to kafka cluster from seperate ips modify listeners separately for broker and controller.
add advertised listeners from different ips.

What alternatives have you considered?

we created kafka cluster with manifests to handle this problem.

sample of broker manifest: 

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  lables:
    app.kubernetes.io/name: kafka
    app.kubernetes.io/version: 3.7.1
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: 172.16.1.68:8082/docker.io/bitnami/kafka:3.7.1-debian-12-r4
          resources:
            limits:
              memory: "8Gi"
              cpu: "4"
            requests:
              memory: "4Gi"
              cpu: "2"
          ports:
            - containerPort: 9096
              name: plaintext
            - containerPort: 9095
              name: controller
            - containerPort: 9092
              name: exone
            - containerPort: 9093
              name: extwo
            - containerPort: 9094
              name: exthree
          env:
            - name: KAFKA_CFG_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
            - name: KAFKA_CFG_PROCESS_ROLES
              value: "controller,broker"
            - name: KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
              value: "0@kafka-0.kafka-headless.backend.svc.cluster.local:9095,1@kafka-1.kafka-headless.backend.svc.cluster.local:9095,2@kafka-2.kafka-headless.backend.svc.cluster.local:9095"
            - name: KAFKA_KRAFT_CLUSTER_ID
              value: "abcdefghijklmnopqrstuv"
            - name: KAFKA_CFG_LISTENERS
              value: "EXONE://:9092,EXTWO://:9093,EXTHREE://:9094,CONTROLLER://:9095,PLAINTEXT://:9096"
            - name: KAFKA_CFG_ADVERTISED_LISTENERS
              value: "EXONE://192.168.1.201:9092,EXTWO://192.168.1.201:9093,EXTHREE://192.168.1.201:9094,PLAINTEXT://:9096"
            - name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
              value: "EXONE:PLAINTEXT,EXTWO:PLAINTEXT,EXTHREE:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
            - name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES
              value: "CONTROLLER"
            - name: KAFKA_CFG_INTER_BROKER_LISTENER_NAME
              value: "PLAINTEXT"
            - name: KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR
              value: "2"
            - name: KFAKA_CFG_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
            - name: KAFKA_CFG_LOG_RETENTION_HOURS
              value: "24"
          volumeMounts:
            - name: kafka-data
              mountPath: /bitnami/kafka

  volumeClaimTemplates:
    - metadata:
        name: kafka-data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 2000Gi
        storageClassName: local-path
---

sample of services: 

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  labels:
    app: kafka
spec:
  ports:
    - port: 9096
      name: plaintext
    - port: 9095
      name: controller
    - port: 9092
      name: exone
    - port: 9093
      name: extwo
    - port: 9094
      name: exthree
  clusterIP: None
  selector:
    app: kafka

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker
  labels:
    app: kafka
spec:
  type: NodePort
  ports:
    - port: 9092
      name: exone
      targetPort: 9092
      nodePort: 30092
      protocol: TCP
    - port: 9093
      name: extwo
      targetPort: 9093
      nodePort: 30093
      protocol: TCP
    - port: 9094
      name: exthree
      targetPort: 9094
      nodePort: 30094
      protocol: TCP
  selector:
    app: kafka
carrodher commented 1 month ago

The issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a specific scenario that is not easy to reproduce on our side.

If you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

sinamohsenifar commented 1 month ago

Thank you for your response. I understand that the issue might be specific to our configuration. I am considering contributing a solution and will review the contributing guidelines. Meanwhile, could you provide any additional insights or recommendations on how to configure the listeners in a similar environment effectively? Your guidance will be greatly appreciated.

javsalgar commented 1 month ago

Hi,

I'm not sure if I understood correctly, but you mention that you want to have separate IPs, right? Would the external access option work for your use case?

## External Access to Kafka brokers configuration
##
externalAccess:
  ## @param externalAccess.enabled Enable Kubernetes external cluster access to Kafka brokers
  ##
  enabled: false
sinamohsenifar commented 1 month ago

my request is that let us to define list of additional listeners and advertised listeners, because some times our clienst cant see kafka or kubernetes ips directly.

migruiz4 commented 1 week ago

Hi @sinamohsenifar,

I'm sorry for the late response. The bitnami/kafka chart already has a feature that allows users to add additional listeners or even override the default ones as desired:

See the 'listeners' section of the values.yaml:

listeners:
  ## @param listeners.client.name Name for the Kafka client listener
  ## @param listeners.client.containerPort Port for the Kafka client listener
  ## @param listeners.client.protocol Security protocol for the Kafka client listener. Allowed values are 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL' and 'SSL'
  ## @param listeners.client.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required'
  client:
    containerPort: 9092
    protocol: SASL_PLAINTEXT
    name: CLIENT
    sslClientAuth: ""
  ## @param listeners.controller.name Name for the Kafka controller listener
  ## @param listeners.controller.containerPort Port for the Kafka controller listener
  ## @param listeners.controller.protocol Security protocol for the Kafka controller listener. Allowed values are 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL' and 'SSL'
  ## @param listeners.controller.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required'
  ## Ref: https://cwiki.apache.org/confluence/display/KAFKA/KIP-684+-+Support+mutual+TLS+authentication+on+SASL_SSL+listeners
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: SASL_PLAINTEXT
    sslClientAuth: ""
  ## @param listeners.interbroker.name Name for the Kafka inter-broker listener
  ## @param listeners.interbroker.containerPort Port for the Kafka inter-broker listener
  ## @param listeners.interbroker.protocol Security protocol for the Kafka inter-broker listener. Allowed values are 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL' and 'SSL'
  ## @param listeners.interbroker.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.authType for this listener. Allowed values are 'none', 'requested' and 'required'
  interbroker:
    containerPort: 9094
    protocol: SASL_PLAINTEXT
    name: INTERNAL
    sslClientAuth: ""
  ## @param listeners.external.containerPort Port for the Kafka external listener
  ## @param listeners.external.protocol Security protocol for the Kafka external listener. . Allowed values are 'PLAINTEXT', 'SASL_PLAINTEXT', 'SASL_SSL' and 'SSL'
  ## @param listeners.external.name Name for the Kafka external listener
  ## @param listeners.external.sslClientAuth Optional. If SASL_SSL is enabled, configure mTLS TLS authentication type. If SSL protocol is enabled, overrides tls.sslClientAuth for this listener. Allowed values are 'none', 'requested' and 'required'
  external:
    containerPort: 9095
    protocol: SASL_PLAINTEXT
    name: EXTERNAL
    sslClientAuth: ""
  ## @param listeners.extraListeners Array of listener objects to be appended to already existing listeners
  ## E.g.
  ## extraListeners:
  ##  - name: CUSTOM
  ##    containerPort: 9097
  ##    protocol: SASL_PLAINTEXT
  ##    sslClientAuth: ""
  ##
  extraListeners: []
  ## NOTE: If set, below values will override configuration set using the above values (extraListeners.*, controller.*, interbroker.*, client.* and external.*)
  ## @param listeners.overrideListeners Overrides the Kafka 'listeners' configuration setting.
  ## @param listeners.advertisedListeners Overrides the Kafka 'advertised.listener' configuration setting.
  ## @param listeners.securityProtocolMap Overrides the Kafka 'security.protocol.map' configuration setting.
  overrideListeners: ""
  advertisedListeners: ""
  securityProtocolMap: ""

I think the value you may be looking for would be extraListeners and/or advertisedListeners/overrideListeners.