search

thin-edge / thin-edge.io

The open edge framework for lightweight IoT devices
https://thin-edge.io
Apache License 2.0
223 stars 55 forks source link

Use REST for measurement collections exceeding the mqtt-limit of 16K #1415

Open toewsar opened 2 years ago

toewsar commented 2 years ago

For measurement collection below 16K its perfect to send via mqtt to use less data traffic. For measurement collections above 16K thats not possible, the mqtt of c8y is limited to 16K messages.

It would be great that thin-edge.io transfers measurement collections larger than 16K automatically via REST.

didier-wenzek commented 2 years ago

I really want to avoid adding this feature - at least for now.

The reason is that this introduces a lot of internal complexity. Not because one has then to deal with two protocols (MQTT and HTTP) but because these protocols are not using the same communication pattern (pub/sub vs req/res). We experienced many issues because of this for events.

However, we are starting a refactoring that will help. So this could be reconsidered - say in 3 months.

Meantime, it would be good to understand the true need here. Why sending several messages would not work? What about having thin-edge cutting large inputs to fit Cumulocity constraints?

toewsar commented 2 years ago

Good point. In my optionon, a better solution would be to split the large collection of measurements into several parts, so that we wouldn't lose the buffer function (in case of connection loss) of mosquitto. Either way, in cumulocity the costs for measured values are settled in 4k blocks.