Example using kubernetes service type: NodePort

[jarojasm95]

Hello,

What is the appropriate configuration for exposing kafka-proxy over a NodePort service? I've looked at https://github.com/grepplabs/kafka-proxy/issues/30 and https://github.com/grepplabs/kafka-proxy/issues/50 but have been unable to get it working.

NOTE: Both kafka-proxy and kafka are running inside kubernetes (using kind). I am using strimzi for deploying kafka.

Thanks!

[everesio]

Hi, could you provide your current kind and strimzi setup ? Kind regards, Michal

[jarojasm95]

Hey @everesio!, here's my configuration: Kind:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30005  # trying to use this port to expose kafka-proxy
    hostPort: 30005

strimzi:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kueue
spec:
  kafka:
    version: 2.6.0
    replicas: 1
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "2.6"
      default.replication.factor: 1
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

[everesio]

Hi @jarojasm95,

please try this out

1. pods after kafka strimzi operator and kafka CRD were installed in kafka namespace

$ kubectl get pods -A                         
NAMESPACE            NAME                                                        READY   STATUS    RESTARTS   AGE
kafka                kueue-entity-operator-6b9d789c88-zl9vk                      3/3     Running   0          21h
kafka                kueue-kafka-0                                               1/1     Running   0          21h
kafka                kueue-zookeeper-0                                           1/1     Running   0          21h
kafka                strimzi-cluster-operator-54ff55979f-tsdsl                   1/1     Running   0          21h
kube-system          coredns-f9fd979d6-h9jvb                                     1/1     Running   0          22h
kube-system          coredns-f9fd979d6-rbsv4                                     1/1     Running   0          22h
kube-system          etcd-strimzi-kafka-proxy-control-plane                      1/1     Running   0          22h
kube-system          kindnet-pbx44                                               1/1     Running   0          22h
kube-system          kube-apiserver-strimzi-kafka-proxy-control-plane            1/1     Running   0          22h
kube-system          kube-controller-manager-strimzi-kafka-proxy-control-plane   1/1     Running   0          22h
kube-system          kube-proxy-nwkgm                                            1/1     Running   0          22h
kube-system          kube-scheduler-strimzi-kafka-proxy-control-plane            1/1     Running   0          22h
local-path-storage   local-path-provisioner-78776bfc44-zdvrk                     1/1     Running   0          22h

2. check kafka config

   
   $ kubectl exec -it -n kafka kueue-kafka-0 -- ps -ef
   UID          PID    PPID  C STIME TTY          TIME CMD
   kafka          1       0  0 Dec12 ?        00:00:01 /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
   ...

$ kubectl exec -it -n kafka kueue-kafka-0 -- cat /tmp/strimzi.properties ... listeners=REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093 advertised.listeners=REPLICATION-9091://kueue-kafka-0.kueue-kafka-brokers.kafka.svc:9091,PLAIN-9092://kueue-kafka-0.kueue-kafka-brokers.kafka.svc:9092,TLS-9093://kueue-kafka-0.kueue-kafka-brokers.kafka.svc:9093 listener.security.protocol.map=REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL ...

PLAINTEXT listener is advertised as `kueue-kafka-0.kueue-kafka-brokers.kafka.svc:9092`

3. deploy kafka-proxy and use 30005 as node port which in you kind config is exposed as host port

---

apiVersion: apps/v1 kind: Deployment metadata: name: kafka-proxy spec: replicas: 1 selector: matchLabels: app: kafka-proxy template: metadata: labels: app: kafka-proxy spec: containers:

• name: kafka-proxy image: grepplabs/kafka-proxy:v0.2.8 args:
  • 'server'
  • '--log-level=debug'

  • '--bootstrap-server-mapping=kueue-kafka-0.kueue-kafka-brokers.kafka.svc:9092,0.0.0.0:30005'

    apiVersion: v1 kind: Service metadata: name: kafka-proxy spec: type: NodePort selector: app: kafka-proxy ports:

    • protocol: TCP port: 30005 targetPort: 30005 nodePort: 30005

4. Now you can use kafka consumer / producer from your host

bin/kafka-console-producer.sh --broker-list 127.0.0.1:30005 --topic my-topic
bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:30005 --topic my-topic --from-beginning