KafkaUsers support over Istio MTLS

[2phost]

Problem I have a kafka cluster running over MTLS using Istio. Everything is working as expected, except the KafkaUsers.

Requested feature I would like to see the possibility of using ACLs when using Istio MTLS.

[baluchicken]

This is available in Supertubes, our commercial Kafka product. It’s a fairly complex feature, which includes a KafkaPrincipalBuilder, the Kafka Envoy filter and some other tweaks to make it seamlessy work out of the box with existing ACLs. You can read about in more details in this post or check out the Supertubes docs.

[stoader]

@2phost closing this issue as there is support for KafkaUsers over Istio mTLS in Supertubes. You can read more on it here: Kafka ACLs on Kubernetes over Istio mTLS

[satishmane]

Hi @2phost

my 3 pod kafka cluster fails when istio mtls is enabled by making namespace STRICT. Can you please refer my below ticket.

Can you please share what steps did you take to make kafka work with istio mtls. thanks

https://github.com/istio/istio/issues/26791

[baluchicken]

Hi @satishmane your error can be caused by many things. We need more information about your settings. Can you please share your KafkaCluster CR?

[satishmane]

thanks @baluchicken

kafka is installed using charts / statefulsets

=== stateful set

apiVersion: apps/v1 kind: StatefulSet metadata: generation: 1 labels: app: kafka chart: kafka-2.1.0 heritage: Tiller release: infra-kafka name: infra-kafka namespace: infra spec: podManagementPolicy: Parallel replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: app: kafka chart: kafka-2.1.0 heritage: Tiller release: infra-kafka serviceName: infra-kafka-headless template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "true" sidecar.istio.io/inject: "true" sidecar.istio.io/rewriteAppHTTPProbers: "true" labels: app: kafka chart: kafka-2.1.0 heritage: Tiller release: infra-kafka name: infra-kafka spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution:

• podAffinityTerm: labelSelector: matchLabels: app: kafka release: infra-kafka topologyKey: failure-domain.beta.kubernetes.io/zone weight: 50 requiredDuringSchedulingIgnoredDuringExecution:
• labelSelector: matchLabels: app: kafka release: infra-kafka topologyKey: kubernetes.io/hostname containers:
  • env:
    • name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP
    • name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name
    • name: KAFKA_ZOOKEEPER_CONNECT value: infra-kafka-zookeeper
    • name: KAFKA_PORT_NUMBER value: "9092"
    • name: KAFKA_LISTENERS value: PLAINTEXT://:$(KAFKA_PORT_NUMBER)
    • name: KAFKA_ADVERTISED_LISTENERS value: PLAINTEXT://$(MY_POD_NAME).infra-kafka-headless.infra:$(KAFKA_PORT_NUMBER)
    • name: ALLOW_PLAINTEXT_LISTENER value: "yes"
    • name: KAFKA_BROKER_ID value: "-1"
    • name: KAFKA_DELETE_TOPIC_ENABLE value: "true"
    • name: KAFKA_HEAP_OPTS value: -Xmx1025m -Xms1025m
    • name: KAFKA_LOG_FLUSH_INTERVAL_MESSAGES value: "10000"
    • name: KAFKA_LOG_FLUSH_INTERVAL_MS value: "1000"
    • name: KAFKA_LOG_RETENTION_BYTES value: "1073741824"
    • name: KAFKA_LOG_RETENTION_CHECK_INTERVALS_MS value: "300000"
    • name: KAFKA_LOG_RETENTION_HOURS value: "1"
    • name: KAFKA_MAX_MESSAGE_BYTES value: "1000012"
    • name: KAFKA_SEGMENT_BYTES value: "1073741824"
    • name: KAFKA_LOGS_DIRS value: /opt/kafka/data
    • name: KAFKA_NUM_IO_THREADS value: "8"
    • name: KAFKA_NUM_NETWORK_THREADS value: "3"
    • name: KAFKA_NUM_PARTITIONS value: "8"
    • name: KAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR value: "1"
    • name: KAFKA_SOCKET_RECEIVE_BUFFER_BYTES value: "102400"
    • name: KAFKA_SOCKET_REQUEST_MAX_BYTES value: "104857600"
    • name: KAFKA_SOCKET_SEND_BUFFER_BYTES value: "102400"
    • name: KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS value: "6001" image: kafka:2.1.0-1 imagePullPolicy: Always livenessProbe: failureThreshold: 3 initialDelaySeconds: 70 periodSeconds: 10 successThreshold: 1 tcpSocket: port: tcp-kafka timeoutSeconds: 5 name: infra-kafka ports:
    • containerPort: 9092 name: tcp-kafka protocol: TCP readinessProbe: failureThreshold: 6 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 tcpSocket: port: tcp-kafka timeoutSeconds: 5 resources: limits: cpu: "2" memory: 8Gi requests: cpu: "1" memory: 4Gi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /opt/kafka name: data
    • mountPath: /opt/kafka/conf/server.properties name: kafka-config subPath: server.properties
  • command:
    • java
    • -XX:+UnlockExperimentalVMOptions
    • -XX:+UseCGroupMemoryLimitForHeap
    • -XX:MaxRAMFraction=1
    • -XshowSettings:vm
    • -jar
    • jmx_prometheus_httpserver.jar
    • "5556"
    • /etc/jmx-kafka/jmx-kafka-prometheus.yml image: kafka-prometheus-jmx-exporter:1.0.1 imagePullPolicy: Always name: jmx-exporter ports:
    • containerPort: 5556 protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
    • mountPath: /etc/jmx-kafka name: jmx-config dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 1001 runAsUser: 1001 terminationGracePeriodSeconds: 30 volumes:
  • configMap: defaultMode: 420 name: infra-kafka-jmx-configuration name: jmx-config
  • configMap: defaultMode: 420 name: infra-kafka-configuration name: kafka-config updateStrategy: rollingUpdate: partition: 0 type: RollingUpdate volumeClaimTemplates:
    • metadata: creationTimestamp: null name: data spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 300Gi volumeMode: Filesystem status: phase: Pending

====== headless service

apiVersion: v1 kind: Service metadata: labels: app: kafka chart: kafka-2.1.0 heritage: Tiller release: infra-kafka name: infra-kafka-headless namespace: infra spec: clusterIP: None ports:

• name: tcp-kafka port: 9092 protocol: TCP targetPort: tcp-kafka selector: app: kafka release: infra-kafka sessionAffinity: None type: ClusterIP status: loadBalancer: {}

==== cluster ip service

apiVersion: v1 kind: Service metadata: labels: app: kafka chart: kafka-2.1.0 heritage: Tiller release: infra-kafka name: infra-kafka namespace: infra spec: clusterIP: 10.100.125.188 ports:

• name: tcp-kafka port: 9092 protocol: TCP targetPort: tcp-kafka selector: app: kafka release: infra-kafka sessionAffinity: None type: ClusterIP status: loadBalancer: {}

[2phost]

Hi @2phost

my 3 pod kafka cluster fails when istio mtls is enabled by making namespace STRICT. Can you please refer my below ticket.

Can you please share what steps did you take to make kafka work with istio mtls. thanks

istio/istio#26791

Hi @satishmane,

I assume that you are using the Operator from Banzai. I have made several modifications in order to get it working as I want (specific listeners for multi clusters and many others), but it should work fine out of the box.

May I ask you to share your Kafka Custom Resource? The error can be caused by many things, and the root cause can be in your cluster configuration.

Both, stateful set and services are created by the Operator, so we really need to check your CR specification.