mqtt-vue-hook to Confluent/Kafka with SASL_SSL requirement

[donniekerr]

I have a node.js client that connects to Confluent Cloud using the below options. I'd like to be able to publish events in my Vue UI to Confluent Cloud and also subscribe to topics. Kafkajs only works in Node. Looks like your hook can help me with it. I just can't tell if you can pass in these options and what the options object should look like to do SASL. Can it work? Required connection configs for Kafka producer, consumer, and admin bootstrap.servers=pkc-57jzz.southcentralus.azure.confluent.cloud:9092 security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username=xxxx sasl.password=xxxx

[tommy44458]

Hi @donniekerr,

Thanks for your question. Kafka isn't designed to be publicly accessible to my best knowledge. Have you considered the MQTT or HTTP proxy for your Kafka on the server side? I think you can refer to the following information:

https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot https://github.com/confluentinc/kafka-rest

[donniekerr]

Yes, I am trying to connect to MQTT Proxy for Confluent Cloud vs going through a separate broker. I am told it will work but it needs the SSL_SASL and PLAIN mechanism options.

On Jul 18, 2022, at 1:23 PM, tommy44458 @.***> wrote:

 Hi @donniekerr,

Thanks for your question. Kafka isn't designed to be publicly accessible to my best knowledge. Have you considered the MQTT or HTTP proxy for your Kafka on the server side? I think you can refer to the following information:

https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot https://github.com/confluentinc/kafka-rest

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[tommy44458]

Your configuration option seems for connecting to your Kafka. The MQTT connection option would be like "(wss, tls):x.x.x.x:(8083, 8883)" if you build the MQTT proxy by https://docs.confluent.io/platform/current/kafka-mqtt/configuration_options.html. Also, you can refer to https://docs.confluent.io/platform/current/kafka-mqtt/security-settings.html#mqtt-proxy-security-settings to select the protocol between your MQTT Proxy and MQTT clients(Vue UI).

[tommy44458]

I have a node.js client that connects to Confluent Cloud using the below options. I'd like to be able to publish events in my Vue UI to Confluent Cloud and also subscribe to topics. Kafkajs only works in Node. Looks like your hook can help me with it. I just can't tell if you can pass in these options and what the options object should look like to do SASL. Can it work? Required connection configs for Kafka producer, consumer, and admin bootstrap.servers=pkc-57jzz.southcentralus.azure.confluent.cloud:9092 security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username=xxxx sasl.password=xxxx

I'm not sure if this works, but maybe you can try the option below:

app.use(mqttVueHook, `wss://pkc-57jzz.southcentralus.azure.confluent.cloud:9092`, {
    clean: false,
    keepalive: 60,
    clientId: `mqtt_client_${Math.random().toString(16).substring(2, 10)}`,
    username: xxxx,
    password: xxxx,
})

[donniekerr]

Tried it. All I get is failed with no reason.
I'm working this from both side, the client and mqtt proxy. https://forum.confluent.io/t/use-mqtt-proxy-in-confluent-cloud/5561 Seems I must has to explicitly pass in the SASL_SSL and PLAIN mechanism but I looked and MQTT.js docs and can seem to find how to pass it to the Client constructor. It says the MQTT Proxy is supposed to work over Websockets.

My goal here is to not have to add a waterstream or HiveMQ, etc. broker in between my IoT devices and Kafka and in between the browser and Kafka. Trying to keep the architecture as simple as possible and as cheap as possible.

[donniekerr]

`// protocol = 'wss', 'ws', 'mqtt', ... const protocol = 'wss' // host = ip or domain const host = 'pkc-57jzz.southcentralus.azure.confluent.cloud' const port = 9092 // port = 8083, 1883, ... import mqttVueHook from 'mqtt-vue-hook' // app.use(mqttVueHook, options)

const app = createApp(App) .use(IonicVue) .use(router) .use(mqttVueHook, ${protocol}://${host}:${port}, { clean: false, keepalive: 60, clientId: mqtt_client_${Math.random().toString(16).substring(2, 10)}, username: '2PXXXX', password: 'quXXXX'

})` I tried mqtt protocol too and it gets converted to ws:// . And I used wss.

[tommy44458]

Your configuration option seems for connecting to your Kafka. The MQTT connection option would be like "(wss, tls):x.x.x.x:(8083, 8883)" if you build the MQTT proxy by https://docs.confluent.io/platform/current/kafka-mqtt/configuration_options.html. Also, you can refer to https://docs.confluent.io/platform/current/kafka-mqtt/security-settings.html#mqtt-proxy-security-settings to select the protocol between your MQTT Proxy and MQTT clients(Vue UI).

Hi @donniekerr,

I think your Kafka-MQTT-dev.properties would be wrong. Could you share your dev-config for building the MQTT proxy? The correct "dev.properties" would be like below:

topic.regex.list=temperature:.*temperature, brightness:.*brightness
# MQTT proxy
listeners=0.0.0.0:1883
# bootstrap server url
bootstrap.servers=pkc-57jzz.southcentralus.azure.confluent.cloud:9092
# security protocol
producer.security.protocol=SASL_SSL
# sasl mechanism
producer.sasl.mechanism=PLAIN
# sasl jaas config
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule   required username='api_key_from_confluent_cloud'   password='api_secret_from_confluent_cloud';
# confluent cloud refuses to create topic if you use default replication factor of 1
confluent.topic.replication.factor=3

Then, you can try the following option to connect your MQTT proxy:

app.use(mqttVueHook, `ws://pkc-57jzz.southcentralus.azure.confluent.cloud:1883`, {
    clean: false,
    keepalive: 60,
    clientId: `mqtt_client_${Math.random().toString(16).substring(2, 10)}`,
    username: xxxx,
    password: xxxx,
})

[donniekerr]

At the same time I’m taking to you I have been trying to find out how the MQTT Proxy works in the cloud version of Confluent, not the on-prem platform. My assumption was that it could talk directly to a Confluent Cloud cluster. That assumption was wrong, as I just found out.
I may as well setup a broker in front of Confluent if I have to stand up another server to run the proxy. Then I know your hook can talk to a true broker.

Sorry if I wasted your time. I am learning. We can close this for now. Thank you, Donnie

On Jul 18, 2022, at 4:50 PM, tommy44458 @.***> wrote:

Your configuration option seems for connecting to your Kafka. The MQTT connection option would be like "(wss, tls):x.x.x.x:(8083, 8883)" if you build the MQTT proxy by https://docs.confluent.io/platform/current/kafka-mqtt/configuration_options.html https://docs.confluent.io/platform/current/kafka-mqtt/configuration_options.html. Also, you can refer to https://docs.confluent.io/platform/current/kafka-mqtt/security-settings.html#mqtt-proxy-security-settings https://docs.confluent.io/platform/current/kafka-mqtt/security-settings.html#mqtt-proxy-security-settings to select the protocol between your MQTT Proxy and MQTT clients(Vue UI).

Hi @donniekerr https://github.com/donniekerr,

I think your Kafka-MQTT-dev.properties would be wrong. Could you share your dev-config for building the MQTT proxy? The correct "dev.properties" would be like below:

topic.regex.list=temperature:.temperature, brightness:.brightness

MQTT proxy

listeners=0.0.0.0:1883

bootstrap server url

bootstrap.servers=pkc-57jzz.southcentralus.azure.confluent.cloud:9092

security protocol

producer.security.protocol=SASL_SSL

sasl mechanism

producer.sasl.mechanism=PLAIN

sasl jaas config

producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='api_key_from_confluent_cloud' password='api_secret_from_confluent_cloud';

confluent cloud refuses to create topic if you use default replication factor of 1

confluent.topic.replication.factor=3 Then, you can try the following option to connect your MQTT proxy:

app.use(mqttVueHook, ws://pkc-57jzz.southcentralus.azure.confluent.cloud:1883, { clean: false, keepalive: 60, clientId: mqtt_client_${Math.random().toString(16).substring(2, 10)}, username: xxxx, password: xxxx, }) — Reply to this email directly, view it on GitHub https://github.com/tommy44458/mqtt-vue-hook/issues/1#issuecomment-1188355839, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZJXIV7XMIVXZMVSIRRMWDVUXGRZANCNFSM53UDTUSA. You are receiving this because you were mentioned.

[donniekerr]

closing this because it can't work with Confluent Cloud cluster without a separate proxy server or broker server.