Bring your own Kafka example

[ra014620]

I've got the mqtt.kafka.broker example working as-is but I'm have difficulty figuring out exactly how to use my existing Kafka broker. In the zilla.yaml file I've tried replacing the host and port under tcp_client0 but that didn't work. Is there an example that shows how to do this? Thanks.

[vordimous]

Hi @ra014620, I am sorry you aren't able to connect. Can you tell me where the Kafka you are trying to connect to lives?

In the mean time here is some more info on how to connect to different Kafka installs.

[ra014620]

It’s a standalone outside the cluster. And thanks for the link. That is what I was following. My case is the PLAINTEXT one. I can connect to the mqtt server but it seems that the proxy can’t find the my Kafka server.

[vordimous]

Ok, thank you. I have been working on replicating this locally and will let you know when I have an update.

[vordimous]

@ra014620 I was able to get the mqtt.kafka.broker example to work locally. Two think you will need to do

1. Update the Helm chart version in setup.sh script:

VERSION=0.9.54

2. Pull the latest version of Zilla:

docker pull ghcr.io/aklivity/zilla:latest

Doing both of those should fix any issues on the Zilla side. If there are still issues connecting to your plaintext Kafka I am using an external kafka setup that looks like this:

version: "3"
services:
  kafka:
    image: docker.io/bitnami/kafka:latest
    container_name: kafka
    restart: unless-stopped
    ports:
      - 9092:9092
      - 29092:9092
    environment:
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_CFG_NODE_ID: "1"
      KAFKA_CFG_BROKER_ID: "1"
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: "1@127.0.0.1:9093"
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "CLIENT:PLAINTEXT,INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_CFG_LOG_DIRS: "/tmp/logs"
      KAFKA_CFG_PROCESS_ROLES: "broker,controller"
      KAFKA_CFG_LISTENERS: "CLIENT://:9092,INTERNAL://:29092,CONTROLLER://:9093"
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_CFG_ADVERTISED_LISTENERS: "CLIENT://host.docker.internal:9092,INTERNAL://host.docker.internal:29092"
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: "true"

  kafka-init:
    image: docker.io/bitnami/kafka:latest
    container_name: temp-kafka-init-zq
    command: 
      - "/bin/bash"
      - "-c"
      - |
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 --create --if-not-exists --topic mqtt-messages
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 --create --if-not-exists --topic mqtt-retained --config cleanup.policy=compact
        /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server kafka:29092 --create --if-not-exists --topic mqtt-sessions --config cleanup.policy=compact
    depends_on:
      - kafka
    init: true

Don't forget to create the Kafka topics.

[vordimous]

There is also a simpler version of an MQTT Kafka broker in the docs if it helps: https://docs.aklivity.io/zilla/latest/tutorials/mqtt/mqtt-intro.html

[ra014620]

Thank you. I'm trying it now and will let you know how it goes.

[ra014620]

What does your zilla.yaml look like? Under the tcp_client0 section, the host and port should be the hostname and port of my external Kafka cluster, correct?

[vordimous]

What does your zilla.yaml look like?

I am using the local docker DNS but it should be replicating the same use case.

  kafka_client0:
    type: kafka
    kind: client
    exit: tcp_client0
  tcp_client0:
    type: tcp
    kind: client
    options:
      host: host.docker.internal
      port: 29092
    routes:
      - when:
          - cidr: 0.0.0.0/0

Under the tcp_client0 section, the host and port should be the hostname and port of my external Kafka cluster, correct?

Yes, it should just be the same location any other application would point.

[vordimous]

I went ahead and updated the ports to all use the same port and was able to connect successfully so my Kafka config has:

KAFKA_CFG_ADVERTISED_LISTENERS: "CLIENT://host.docker.internal:9092,INTERNAL://host.docker.internal:9092"

and my zilla.yaml

    options:
      host: host.docker.internal
      port: 9092

[vordimous]

Are you getting a connection refused error or does zilla start and your attempt to connect over MQTT :1883 just hanging?

if the :1883 connection is hanging the most common issue is all of the necessary topics aren't created.

[ra014620]

I used your Kafka docker-compose and got the mqtt-proxy working. Must be something off with my other Kafka config. Thanks for your help on this.