[bitnami/kafka] After Kafka jmx monitoring is enabled, Prometheus is unable to obtain metrics

[junlintianxiazhifulinzhongguo]

Name and Version

bitnami/kafka:3.7.0-debian-12-r3

What architecture are you using?

None

What steps will reproduce the bug?

Are you using any custom parameters or values?

image:
  repository: bitnami/kafka
  tag: 3.7.0-debian-12-r3

listeners:
  client:
    containerPort: 9092
    protocol: SASL_PLAINTEXT
    name: CLIENT

  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: SASL_PLAINTEXT

  interbroker:
    containerPort: 9094
    protocol: SASL_PLAINTEXT
    name: INTERNAL

  external:
    containerPort: 9095
    protocol: SASL_PLAINTEXT
    name: EXTERNAL

sasl:
  enabledMechanisms: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
  interBrokerMechanism: PLAIN
  controllerMechanism: PLAIN

  interbroker:
    user: inter_broker_user
    password: "wTmXMmWMGwXU"

  controller:
    user: controller_user
    password: "wTmXMmWMGwXU"

  client:
    users:
    - admin
    passwords: "wTmXMmWMGwXU"

extraEnvVars:
- name: TZ
  value: "Asia/Shanghai"

extraVolumes:
- name: host-time
  hostPath:
    path: /etc/localtime
    type: ''
- name: kafka-client-config-properties
  configMap:
    name: kafka-client-config-properties
    defaultMode: 420

extraVolumeMounts:
- name: host-time
  readOnly: true
  mountPath: /etc/localtime
- name: kafka-client-config-properties
  mountPath: /bitnami/kafka/config/kafka.client.config.properties
  subPath: kafka.client.config.properties

controller:
  heapOpts: -Xmx1024m -Xms1024m
  resources:
    limits:
      cpu: '1'
      memory: 2Gi
    requests:
      cpu: 300m
      memory: 800Mi
  persistence:
    size: 10Gi

externalAccess:
  enabled: true

  controller:
    service:
      type: NodePort
      nodePorts:
      - 32701
      - 32702
      - 32703
      useHostIPs: true

kraft:
  enabled: true

metrics:
  kafka:
    enabled: true
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi

  jmx:
    enabled: true
    kafkaJmxPort: 5555
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi

  serviceMonitor:
    enabled: true
    namespace: "monitor"
    labels:
      release: kube-prometheus-stack
    jobLabel: "kafka-hangli"

  prometheusRule:
    enabled: true
    namespace: "monitor"
    labels:
      release: kube-prometheus-stack

What is the expected behavior?

No response

What do you see instead?

[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]# kubectl get svc -n kafka
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kafka                         ClusterIP   10.233.39.62    <none>        9092/TCP,9095/TCP            24h
kafka-controller-0-external   NodePort    10.233.21.196   <none>        9094:32701/TCP               24h
kafka-controller-1-external   NodePort    10.233.28.122   <none>        9094:32702/TCP               24h
kafka-controller-2-external   NodePort    10.233.33.76    <none>        9094:32703/TCP               24h
kafka-controller-headless     ClusterIP   None            <none>        9094/TCP,9092/TCP,9093/TCP   24h
kafka-jmx-metrics             ClusterIP   10.233.23.242   <none>        5556/TCP                     24h
kafka-metrics                 ClusterIP   10.233.55.78    <none>        9308/TCP                     24h
kafka-ui                      ClusterIP   10.233.19.138   <none>        8080/TCP                     9d
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]# kubectl get pods -n kafka
NAME                             READY   STATUS    RESTARTS      AGE
kafka-controller-0               2/2     Running   0             46m
kafka-controller-1               2/2     Running   0             46m
kafka-controller-2               2/2     Running   0             47m
kafka-exporter-d6476fdbc-m6tv5   1/1     Running   2 (24h ago)   24h
kafka-ui-v1-bb849f55b-ldsbl      1/1     Running   0             4d22h
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]# kubectl get svc -n kafka
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kafka                         ClusterIP   10.233.39.62    <none>        9092/TCP,9095/TCP            24h
kafka-controller-0-external   NodePort    10.233.21.196   <none>        9094:32701/TCP               24h
kafka-controller-1-external   NodePort    10.233.28.122   <none>        9094:32702/TCP               24h
kafka-controller-2-external   NodePort    10.233.33.76    <none>        9094:32703/TCP               24h
kafka-controller-headless     ClusterIP   None            <none>        9094/TCP,9092/TCP,9093/TCP   24h
kafka-jmx-metrics             ClusterIP   10.233.23.242   <none>        5556/TCP                     24h
kafka-metrics                 ClusterIP   10.233.55.78    <none>        9308/TCP                     24h
kafka-ui                      ClusterIP   10.233.19.138   <none>        8080/TCP                     9d
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]#
[root@k8s-master-01 kafka]# curl http://10.233.23.242:5556
curl: (7) Failed to connect to 10.233.23.242 port 5556: Connection timed out

Unable to obtain metrics

Additional information

k8s : v1.23.17

chart: 28.2.1

kafka: 3.7.0

mode: kraft

[javsalgar]

Hi!

Could you try disabling the network policy? Just to ensure that the bug is not there

[junlintianxiazhifulinzhongguo]

@javsalgar Thank you. After disabling the network policy, it returned to normal

[ph311o]

We are facing the same problem. The Network-Policy-Template is not taking into account of metrics.jmx.containerPorts.metrics. I will create a pull request for the issue.

[javsalgar]

Thank you so much! The community will benefit from this