thin-edge / thin-edge.io

The open edge framework for lightweight IoT devices
https://thin-edge.io
Apache License 2.0
221 stars 54 forks source link

Avoid resending all entity registration messages on mapper restart #2314

Closed albinsuresh closed 3 months ago

albinsuresh commented 1 year ago

Is your feature improvement request related to a problem? Please describe.

Since entity registration messages are retained messages on the MQTT broker, every time the broker is started, these messages are re-delivered to the mapper and the mapper will convert those to C8y registration messages and send them again to the cloud, even if they were sent earlier. When there are a large number of entities registered, this leads to unnecessarily resending a lot of messages to the cloud on every mapper startup. Even though duplicate registration messages are ignored by the cloud, it would be better to avoid resending them unnecessarily.

Describe the solution you'd like

When the mapper receives entity registration messages, persist that information on the file system as well, so that on restart, the mapper can reload the previously known state from the file system, compare it against the retained messages that are delivered and only send the differences to the cloud (any new registration messages that may have arrived while the mapper was down).

Describe alternatives you've considered

Whenever the mapper processes a registration message, replace the original registration message on the broker with a copy of it with an additional "@c8y_mapped": "true" flag added to it. When these messages are re-delivered on mapper restart, the ones with this flag can be ignored as they are already processed.

But this approach is more intrusive in the sense that every component subscribed to entity registration messages will get these duplicate messages as well with this additional, which they may not know what to do with. They can just ignore it, but that update itself was unnecessary from their POV.

albinsuresh commented 3 months ago

This feature was implemented in #2522 and can be enabled by setting the c8y.entity_store.clean_start config setting to false.