For example, if on a boat that has sensors like radar, sonar, lidar, or if we took sensor data from a coastal high-frequency radar station, this sensor data needs to be exchanged as fast as possible with the server.
But the same need of messages might already be needed if we simply deal with a very large amount of data, while we only want to transport a tiny bit of it to a frontend like a mobile app, and we would still like to know the status of that frontend. Thus, we need a publish / subscribe exchange which is about the same as with ROS in robotics, but for big data, which should be MQTT then, though doing something with ROS is not excluded when we integrate data from real-time sensor data.
Therefore, we might need MQTT and Apache Kafka. MQTT does not seem to have the ability to handle very high throughput from sensors so again we have to turn to Kafka which can handle high throughput.
Initially it will appear that with the easiness we installed Apache Kafka and MQTT Server, with same easiness we can connect Apache Kafka with MQTT. There are actually many things to learn, think. MQTT is a protocol allowing publish/subscribe exchanges. Multiple implementations of client libraries and brokers are compatible. MQTT only specifies the transport, and the application part like how data will be handled and possibly stored etc are vague. Apache Kafka is a message broker with the focus of storing massive amounts of data, and allowing consumption in real-time or later utilization. Kafka uses own network protocol. We we need to store massive amount of messages, ensure batch processing, we need Kafka. Kafka itself has no built in message priority, security is poor and the heavy protocol makes it difficult to easily use.
For example, if on a boat that has sensors like radar, sonar, lidar, or if we took sensor data from a coastal high-frequency radar station, this sensor data needs to be exchanged as fast as possible with the server.
But the same need of messages might already be needed if we simply deal with a very large amount of data, while we only want to transport a tiny bit of it to a frontend like a mobile app, and we would still like to know the status of that frontend. Thus, we need a publish / subscribe exchange which is about the same as with ROS in robotics, but for big data, which should be MQTT then, though doing something with ROS is not excluded when we integrate data from real-time sensor data. Therefore, we might need MQTT and Apache Kafka. MQTT does not seem to have the ability to handle very high throughput from sensors so again we have to turn to Kafka which can handle high throughput.
Source: How to Connect Apache Kafka With MQTT
Initially it will appear that with the easiness we installed Apache Kafka and MQTT Server, with same easiness we can connect Apache Kafka with MQTT. There are actually many things to learn, think. MQTT is a protocol allowing publish/subscribe exchanges. Multiple implementations of client libraries and brokers are compatible. MQTT only specifies the transport, and the application part like how data will be handled and possibly stored etc are vague. Apache Kafka is a message broker with the focus of storing massive amounts of data, and allowing consumption in real-time or later utilization. Kafka uses own network protocol. We we need to store massive amount of messages, ensure batch processing, we need Kafka. Kafka itself has no built in message priority, security is poor and the heavy protocol makes it difficult to easily use.