Question: numeric value conversion ambiguity

[malessainray]

What do you want to know?

I'm struggling with transmitting values of types other than the directly used protobuf types. For example, to send an int8 I was unable to find any hint in the specification how I should convert the possibly negative int8 value to a uint32 for encoding through protobuf. Could you please point me towards the proper definition of how values must be converted to ensure cross implementation compatibility? As it's ambiguous how values must be converted between datatypes and the specification requires implementations to convert numeric values for transmission I beliebe that the specification shall provide at the very least guidance, or at best specify how values must be converted. Otherwise, cross implementation compatibility cannot be guaranteed, even if all implementations perfectly adhere to the specification.

Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response

Version

None

Accept EFTL Terms

• [X] I agree to the terms of EFTL

[bryce-nakatani]

Sparkplug Metrics contain both the value and the value type. The type allows the Metric's value to be restored on the Host implementation.

As for the value conversion, look at SparkplugBPayloadEncoder.java https://github.com/eclipse/tahu/blob/v1.0.3/java/lib/core/src/main/java/org/eclipse/tahu/message/SparkplugBPayloadEncoder.java (method setMetricValue()), SparkplugBPayloadDecoder.java https://github.com/eclipse/tahu/blob/v1.0.3/java/lib/core/src/main/java/org/eclipse/tahu/message/SparkplugBPayloadDecoder.java (method getMetricValue()), and a Java topic https://docs.oracle.com/javase/specs/jls/se10/html/jls-5.html on "Narrowing Primitive Conversions."

The gist is on the encoder side, values are promoted to the respective data type without losing resolution. So in your case, int8 is just casted to be an int.

On the Host Side, the decoder, knowing the MetricDataType of the Metric, then casts the protobuf value to the appropriate environment data type value by casting. On the decoder, there are data types that are down casted (e.g. int to int8).

On Wed, May 10, 2023 at 12:58 AM malessainray @.***> wrote:

What do you want to know?

I'm struggling with transmitting values of types other than the directly used protobuf types. For example, to send an int8 I was unable to find any hint in the specification how I should convert the possibly negative int8 value to a uint32 for encoding through protobuf. Could you please point me towards the proper definition of how values must be converted to ensure cross implementation compatibility? As it's ambiguous how values must be converted between datatypes and the specification requires implementations to convert numeric values for transmission I beliebe that the specification shall provide at the very least guidance, or at best specify how values must be converted. Otherwise, cross implementation compatibility cannot be guaranteed, even if all implementations perfectly adhere to the specification. Is this related to a Sparkplug Listing request? If so, link the issue from https://github.com/eclipse-sparkplug/sparkplug.listings here.

No response Version

None Accept EFTL Terms

• I agree to the terms of EFTL

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/453, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZRZYUAKSJODBRO4Z3LXFNDB5ANCNFSM6AAAAAAX4LFTWY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[malessainray]

Thanks for your reply! What you describe is what I initially assumed and was successful with for initial tests. But when using C#, simply casting a negative value to an unsigned integer will result in an exception, and since the Sparkplug spec specifies that signed integers must be converted to unsigned protobuf values for the transport layer, and the spec doesn't declare how the conversion must be performed both from a signed value to an unsigned one and vice versa, it is at best a guess to implement a conversion. One might conclude that the spec is missing a section when relying on all implementations behaving identically when casting a negative value to an unsigned integer and back. I suggest that the next iteration of the spec includes such section on value conversion to ensure compatibility across applications. I'd appreciate it if it were possible to get a draft of the above section.

[bryce-nakatani]

I brought up your issue during our most recent specification meeting.

We are pressing forward with the 4.0 specification and don't want to delay this update.

One of the members mentioned his issue https://github.com/eclipse-sparkplug/sparkplug/issues/62 to utilize more of the protobuf's data types. This will be implemented in 4.0 and should provide more clarity into the handling of value. Though this doesn't immediately help you.

In addition, we discussed and agreed about the addition of some kind of documentation to outline how types are handled.

Getting back to your C# situation, when encoding the protobuf stream, there are certain data promotions to fit the protobuf's implemented data types. Using the Sparkplug 3.0 DataTypes listed in 6.4.16, here is how types are converted.

Int8, UInt8, Int16, UInt16, and Int32 are converted to an UInt32 (C# uint). Int64 and UInt64 are converted to an UInt64 (C# ulong). UInt32 can be converted to either a UInt32 or a UInt64 (C# uint or ulong). The 3.0 decoder handles both. DateTime is converted to a UInt64 that represents the milliseconds since Midnight, January 1, 1970, UTC. String, Text, and UUID are strings. Bytes (byte arrays) are byte arrays. For the encoding of the array types Int8Array, Int16Array, Int32Array, Int64Array, UInt8Array, UInt16Array, UInt32Array, UInt64Array, FloatArray, and DoubleArray, these are converter to little endian ordered byte arrays.

As you experienced, casting of specific values (e.g. negative values) can cause issues, especially when the represented value cannot be properly represented by either a uint or a ulong.

I have used a couple of .Net hacks to bit transfer the value and circumvent the bounds checking. Note, there may be more elegant solutions.

// from Int32 to UInt32 uint u32 = BitConverter.ToUInt32(BitConverter.GetBytes(Int32), 0); // from Int64 to UInt64 ulong u64 = BitConverter.ToUInt64(BitConverter.GetBytes(Int64), 0);

On Fri, May 12, 2023 at 8:02 AM malessainray @.***> wrote:

Thanks for your reply! What you describe is what I initially assumed and was successful with for initial tests. But when using C#, simply casting a negative value to an unsigned integer will result in an exception, and since the Sparkplug spec specifies that signed integers must be converted to unsigned protobuf values for the transport layer, and the spec doesn't declare how the conversion must be performed both from a signed value to an unsigned one and vice versa, it is at best a guess to implement a conversion. One might conclude that the spec is missing a section when relying on all implementations behaving identically when casting a negative value to an unsigned integer and back. I suggest that the next iteration of the spec includes such section on value conversion to ensure compatibility across applications. I'd appreciate it if it were possible to get a draft of the above section.

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/453#issuecomment-1545884308, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZUAXLXGX5KQ5OSS45TXFZGIDANCNFSM6AAAAAAX4LFTWY . You are receiving this because you commented.Message ID: @.***>

[SeppPenner]

@bryce-nakatani Is there and ETA when v4 will be available?

[bryce-nakatani]

Based on time and resources, perhaps late 2024. We could always use the help.

On Thu, Dec 7, 2023 at 8:28 AM HansM @.***> wrote:

@bryce-nakatani https://github.com/bryce-nakatani Is there and ETA when v4 will be available?

— Reply to this email directly, view it on GitHub https://github.com/eclipse-sparkplug/sparkplug/issues/453#issuecomment-1845654680, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZRBF667ER2J6VP3NWLYIHVCXAVCNFSM6AAAAAAX4LFTW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVGY2TINRYGA . You are receiving this because you were mentioned.Message ID: @.***>

[SeppPenner]

Based on time and resources, perhaps late 2024. We could always use the help. … On Thu, Dec 7, 2023 at 8:28 AM HansM @.> wrote: @bryce-nakatani https://github.com/bryce-nakatani Is there and ETA when v4 will be available? — Reply to this email directly, view it on GitHub <#453 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEF6AZRBF667ER2J6VP3NWLYIHVCXAVCNFSM6AAAAAAX4LFTW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVGY2TINRYGA . You are receiving this because you were mentioned.Message ID: @.>

Well yeah... I'm already quite busy with keeping my Sparkplug library for C# up-to-date at the moment :D