search

eclipse-sparkplug / sparkplug

Sparkplug
Eclipse Public License 2.0
114 stars 39 forks source link

Question: historical data - corner case #427

Closed pietercardoen closed 1 year ago

pietercardoen commented 1 year ago

What do you want to know?

Dear

We are using the is_historical flag in our sparkplug implementation. We are however not sure how to handle the following corner case: time t1: The EoN sends a NBIRTH message time t2: a metric changes time t3: The EoN sends a new NBIRTH message where several metrics are added or removed time t4: The EoN sends an NDATA message with historical data of the metric change at timestamp t2

Is this valid? What metric alias has to be used? The alias provided by the first birth or the second birth?

Thanks Pieter

Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response

Version

None

Accept EFTL Terms

bryce-nakatani commented 1 year ago

The isHistorical feature is generally intended for use in a store and forward implementation when stored data (during an MQTT Server connection outage) needs to be forwarded in the future when a connection is established to the MQTT Server or back-end application.

time t01 : The Node publishes an NBIRTH with all the current Node metrics. time t02 : Node metrics change, an NDATA containing the value changed metrics is published. time t03 : Node metrics change, an NDATA containing the value changed metrics is published. time t04 : Several metrics are added or removed. Node publishes an NDEATH. time t05 : The Node publishes a new NBIRTH message with the current metric list. Removed or deleted metrics are not included. time t06 : Node metrics change, an NDATA containing the value changed metrics are published. time t07 : Node loses connection to the MQTT Server. If the Node deliberately discontinues the connection, the Node publishes an NDEATH and then disconnects from the MQTT Server. time t08 : Node metrics change, the value changed metrics are stored. time t09 : Node metrics change, the value changed metrics are stored. time t10 : The Node establishes a connection to the MQTT Server. time t11 : The Node publishes an NBIRTH with all the current Node metrics. time t12 : The Node publishes an NDATA message with historical data of the metrics that changed at timestamp t08 (with the stored metrics marked with the isHistorical flag to true in the payload). time t13 : Node metric change. The Node publishes an NDATA message with metric changes and historical data of the metrics that changed at timestamp t09 (with the stored metrics marked with the isHistorical flag to true in the payload). time n...

On Thu, Feb 16, 2023 at 1:19 AM Pieter Cardoen @.***> wrote:

What do you want to know?

Dear

We are using the is_historical flag in our sparkplug implementation. We are however not sure how to handle the following corner case: time t1: The EoN sends a NBIRTH message time t2: a metric changes time t3: The EoN sends a new NBIRTH message where several metrics are added or removed time t4: The EoN sends an NDATA message with historical data of the metric change at timestamp t2

Is this valid? What metric alias has to be used? The alias provided by the first birth or the second birth?

Thanks Pieter Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response Version

None Accept EFTL Terms

  • I agree to the terms of EFTL

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/427, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZQ2EMHVPRLDYJBT6TTWXXWKZANCNFSM6AAAAAAU54YLK4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

asthomas commented 1 year ago

Regarding the metric aliases - the scenario is valid. It would only make sense that you use the metric alias from the second NBIRTH message. The NBIRTH essentially represents a "checkpoint" that establishes the data model and current values at that moment. Every data message after that is an edit relative to that checkpoint (or an edit relative to a previous edit that is relative to that checkpoint). Included in the data model is the alias, which may have changed if metrics have been added or removed. The metric alias from the first NBIRTH message would be invalid. In any case, there is no way for a receiver to associate an NDATA message with any other NBIRTH message besides the most recent.

If you delete the metric between the two NBIRTH messages, you cannot send the historical data since there is no longer a defined metric to which the historical data pertains.

joshwolf-canary commented 1 year ago

That's correct. Only the most recent birth message aliases are valid when sending subsequent NDATA or DDATA messages. When a node gets disconnected from an MQTT server and is buffering data, the data that is buffered should not contain the Sparkplug alias. The Sparkplug alias should be assigned when the node reconnects to the MQTT server and sends the next birth message. So you could keep the Sparkplug alias to tag name mapping in memory but don't include them when buffering the payload (either to disk or memory). If a tag is removed from the device, then any data buffered for that tag should also be removed (referential integrity).