JSON encoding for sparkplug

[Giri-oss]

Hi i was going through sprkplg specification version 2.2 i see many examples are described with JSON but seems actual payload used for communication should be protobuf any specific reason json is not supported as valid format in specification. i understand protobuf format would be very compact but JSON also good for interoperability as it is human readable format and avoids depdency in using specific SDK.

my question is mainly not about either this or that, supporting both formats so that depending on environment apropriate format can be used same can be published as part of birth message so that subscribers are aware of and can be configured to change to different format.

can some one help me if this is some thing evaluated earlier and cautiosly supported only one format (protobuf), wanted to understand background.

[tymoor]

Strongly agree with this! Human readability via standard text encodings has huge benefits beyond that of data savings.

[tymoor]

Protobuf seems like a premature optimization. Simply because it is ‘cool’.

[tymoor]

JSON would be my current choice as well. Human readable, compact, widely supported, and easy to understand in any case.

[wes-johnson]

This has been discussed previously and may become a future variant of Sparkplug - maybe a 'J' version instead of 'B'. I should note this was not really an optimization but rather an early requirement of many of the real world applications that used MQTT. MQTT was designed around bandwidth constrained networks and binary encoding was/is often required to keep payloads small.

[tymoor]

Fair enough, I’m commenting from Phillips 66 - the originator of MQTT with Arlen Nipper and Dr. Andy Stanford-Clark. We are now converting our original Arcom format data (Modbus based, not Protobuf) to JSON and AMQP (0.9.1) mid-flight at the corporate edge for additional features and human readability…

Sent from my iPhone

On May 9, 2022, at 1:04 AM, wes-johnson @.***> wrote:



This has been discussed previously and may become a future variant of Sparkplug - maybe a 'J' version instead of 'B'. I should note this was not really an optimization but rather an early requirement of many of the real world applications that used MQTT. MQTT was designed around bandwidth constrained networks and binary encoding was/is often required to keep payloads small.

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feclipse%2Fsparkplug%2Fissues%2F231%23issuecomment-1120678440&data=05%7C01%7C%7C1e35c6073f274190858808da3181d5bc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637876730947503045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xEUW0LJqalA4WhlEV0zhGCubnxJ%2F8ProMvhW%2FXYBQFo%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAMI4V52WOSWU3NDXN3PUPTVJCTIJANCNFSM5VM52TYA&data=05%7C01%7C%7C1e35c6073f274190858808da3181d5bc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637876730947503045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=C9TqgLXBLpAwNxs3GyZswvhuyIq0dHS%2FlXxgA7PoRu8%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>

[pietercardoen]

Adding json support can be done quite easily as protobuf messages can be translated easily to json. However protobuf may not be seen as a premature optimization but as a solid solution to define packets and a way to construct very compact packets. Using the protobuf file, packet formats are fully defined.

I however doubt if allowing json is a benificial for this specification. One of the main purposes of this specification is interoperability. If multiple data formats would be supported, then supportive tools have to implement all of these data formats to be compatible with all different flavours of devices.

[I-Campbell]

List and or create protobuf <=> Human readable text conversion tools, rather than adding additional encoding formats

"raw UTF8 JSON bytes ==> Human readable text" is slightly less complicated than "raw protobuf bytes ==> Human readable text" That one already has (more than one) tool which does UTF8 JSON Bytes to readable text for you, doesn't mean the solution is to add an additional encoding scheme to all of your communication protocols.

So I propose the solution here, is some tool which does "raw protobuf bytes ==> Human readable text" and vice versa. protoc --encode and protoc --decode does this already. Wireshark does decode. Perhaps what's missing is some neat integration in vscode or eclipse etc, where one can simply copy paste from whichever tool you are getting the json from.

Another argument is, the applications with two supported formats will be twice as big, and have twice as many bugs.

JSON also good for interoperability as it is human readable format ...

I agree with @pietercardoen here, interoperability decreases when you add additional encoding schemes. There are also issues around interoperability between JSON implementations (how are numbers bigger than 2^53 handled?).

*slight exageration

Finally, to quote the sparkplug spec, bold highlighting added: "The purpose of the Sparkplug Specification is to remain true to the original notion of keeping the Topic Namespace and message sizes to a minimum while still making the overall message transactions and session state management between MQTT enabled devices and MQTT SCADA/IIoT applications simple, efficient, easy to understand, and implement."

[matthewGEA]

1. Given that MQTT v5 acknowledges two payload datatypes: 0--unspecified bytes and 1-- UTF-8 Encoded Character Data reference
2. Given that applications exist that translate data from spBv1.0 to other MQTT namespaces
3. Given that the Sparkplug Specification uses JSON formatting to describe payload formats

A statement should be added to the specification that JSON-based payload definitions are normative for applications converting spBv1.0 data into JSON encodings.

This statement will standardize translation applications and allow Sparkplug Information Models to more quickly propagate to existing IT systems/applications that can parse JSON but not protobufs.

[I-Campbell]

Some of the protobuf implementations have a JSON_FORMAT Printer/Parser, eg. https://developers.google.com/protocol-buffers/docs/reference/java

Notably, the official JavaScript does not, but there might be a third party js library which can do that?

So it may be, you could standardise it pretty simply as 'must be equivalent encoding as the Protobuf JSON_FORMAT'

Matthew has changed my mind, I think being able to reach Enterprise applications which have a JSON parsing / encoding library but no way of parsing encoding protobufs outweighs the need to keep sparkplug devices compatible.

Perhaps a translator node could be specified, which takes a JSON formatted topic etc and translates it to the Protobuf format. Perhaps republishing NDATA/edgename to NDATA/edgename_translated, and republishing NCMD/edgename_translated to NCMD/edgename. In this case, *_translated is always the opposite encoding to what the device publishes with. There must be someway of identifying the translator node, so its birth and death can be observed. If #231 is accepted, the translator node should be a separate issue.

[ravisubramanyan]

Is this requirement still planned to be in the 4.0 version of the spec ?

[RickBullotta]

I would strongly encourage that this be reconsidered for 4.0. Many potential consumers of these messages do not have the capability to import the libraries and code needed to deserialize protobuf payloads. That said, I think an alternative solution can be to incorporate JSON format as the default format for the "metric expander" feature being considered. The metric would always be rendered as JSON with a value and timestamp property.

[matthewGEA]

copying over my post in Slack:

The intention of JSON support is not to levy a JSON requirement on plant-floor devices, but to have a standardized path to get that data into enterprise-level systems. Sparkplug aware MQTT Servers should be able to handle this requirement no sweat.

My proposal is as follows:

1. add a standardized mapping to translate between Protobuf and JSON (given all protobuf packets are described in JSON syntax already, this should be a gimmie)
2. include a new namespace (SpJv1.0/#) where the JSON will land (to not break existing deployments)
3. add a requirement for sparkplug aware MQTT servers to include the functionality a user can optionally enable to translate between protobuf and JSON [maybe we should consider decoupling application-level requirements (such as sparkplug aware mqtt servers) separately to allow for more use-cases to be standardized in the future without needing to touch the underlying core protocol/communication mappings specification?]
4. specification to clarify that Edge-of-Network nodes can support either protobuf or JSON or both

[matthewGEA]

Bonus points awarded to combine JSON translation with the "metric expansion" function, as discussed on this issue: https://github.com/eclipse-sparkplug/sparkplug/issues/262

[RickBullotta]

Failure to support JSON encoding in Sparkplug 4.0 would marginalize it completely and make it largely irrelevant for enterprise data integration. It's as simple as that. Enterprise middleware and most apps/platforms that can consume JSON cannot decode specific protobuf formats. Asking them to change is silly. It's a very trivial enhancement to the spec that will pay significant dividends. I'd go so far as to say that without a JSON encoding, Sparkplug becomes completely useless beyond trivial HMI to device integration use cases.

[wz2b]

I kind of agree with @RickBullotta that it's important to have a formalized json object mapping. Without one, people just make up the mapping themselves (I know I do) with no guidance and a slim chance of interoperability. I think it should be added.

[matthewGEA]

I kind of agree with @RickBullotta that it's important to have a formalized json object mapping. Without one, people just make up the mapping themselves (I know I do) with no guidance and a slim chance of interoperability. I think it should be added.

Mapping is step 1, and the Sparkplug Working group has agreed to that.

@RickBullotta correct me if I'm wrong, but you are advocating for more than just a mapping, such as a standard namespace word that indicates that the encoding is JSON, such as "spJv1.0"

A new application type (named Sparkplug Bridge, Sparkplug Gateway, etc) should be created to define a standard method of converting from one spBv1.0 namespace to another spBv1.0 namespace. This same application could be specified to allow conversion from spBv1.0 -> spJv1.0. Then clients using the spJv1.0 namespace would only need to support JSON encoding.

[RickBullotta]

It's 2024. Time for JSON support.