search

drogue-iot / drogue-cloud

Cloud Native IoT
https://drogue.io
Apache License 2.0
113 stars 30 forks source link

Figure out why events had been "late" at the hackathon #344

Open jbtrystram opened 1 year ago

jbtrystram commented 1 year ago

WS does not give an offset when subscribing to kafka

ctron commented 1 year ago

I just peeked into the code and tested this a bit. To me it looks like the WebSocket service doesn't to anything special here, and the default is to use an anonymous group, with not offset reset specified, which should result in "latest". So each new connection should stream from the end of the stream. Using drg stream and the example application (websocket-client) confirm this.

So it looks more like a bottleneck somewhere. We had the gateways streaming data to the cloud, the cloud processing messages (around 15 msgs/s), and the consumer receiving events.

I could imagine that there was an issue with the conference WiFi. Leading to a buildup of messages, either from the gateway to the cloud, or from the cloud to the consumer.

ctron commented 1 year ago

I also think that a scale test might bring some insight here.