[Question] Is there documentation on the kafka storage format?

[martinscheffler]

Is there documentation on the format the messages are stored to in Kafka? Can we consume from the kafka topics with a service? Or should they be seen as internal to TBMQ?

We actually need the MQTT messages in Kafka, so it would make sense to directly consume from a tbmq.msg.app topic. We could also create a MQTT client and make that write to a kafka topic, but that seems wasteful.

[dmytro-landiak]

hi @martinscheffler!

I appreciate your interest in this topic and the question you raised. If you plan to populate the internal TBMQ Kafka topics (e.g., tbmq.msg.app topic) with MQTT messages by publishing those messages from MQTT clients to the broker and later consume them directly with external Kafka consumer, that should be possible. If I misunderstood the question, please correct me or provide more details. In the TBMQ's case, we implement the consumer that fetches the data from topics and then sends the messages to the MQTT subscribers through the MQTT channel.

Regarding the following question:

Is there documentation on the format the messages are stored in Kafka?

Unfortunately, we do not have this documented on our website. Instead, I can navigate you through the main parts of the code/files. You can check the following proto file with messages definition. /common/queue/src/main/proto/queue.proto The one that can be of interest to you is:

message PublishMsgProto {
  string clientId = 1;
  int32 packetId = 2;
  bytes payload = 3;
  bool retain = 4;
  int32 qos = 5;
  string topicName = 6;
  repeated UserPropertyProto userProperties = 7;
  optional MqttPropertiesProto mqttProperties = 8;
}

Which is the ProducerRecord value that is produced to topics tbmq.msg.all or tbmq.msg.app.

If you have any questions, please let me know.

[dmytro-landiak]

Closing as inactive, please reopen in case needed.