search

Azure / Industrial-IoT

Azure Industrial IoT Platform
MIT License
523 stars 214 forks source link

Message mode PubSub with Json message encoding has value in Binary encoding for complex (multilevel structure) data type #2090

Closed nelsonmorais closed 11 months ago

nelsonmorais commented 1 year ago

Describe the bug We have OPC Publisher v2.9.2 running with the following command line parameters --strict --mm=PubSub --me=Json.

We're subscribing to an (internally developed) OPC-UA Server on a node that is a Variable with a DataType of a custom Structure which is made of multiple levels (custom Structure that has fields that are also a custom Structure), and where one of the fields on the inner and deepest level Structures is an optional Field.

Bug: We're publishing the messages to an Azure IoT Hub and observed that the serialization of the Node of this multilevel Structure DataType is not serialized as Json, instead the value is showing the Binary encoding. Other custom Structures with a single level (so none of the Structure Fields point to another custom Structure) are serialized properly.

Here's a sample of the serialization observed on the IoT Hub (see the value of ns=1;s=PostBondingInspection that shows the Binary encoding instead of Json, like the other custom Structures on the same message payload:

{
  "body": {
    "MessageId": "6178ab9f-7b4c-436f-8027-7de6d87b79f1",
    "MessageType": "ua-data",
    "PublisherId": "xxx_module_msft-opc-publisher",
    "DataSetWriterGroup": "EVO3",
    "Messages": [
      {
        "MetaDataVersion": {
          "MajorVersion": 1222365154,
          "MinorVersion": 2465399015
        },
        "MessageType": "ua-deltaframe",
        "DataSetWriterName": "DSW_EVO3",
        "Payload": {
          "ns=1;s=PostBondingInspection": {
            "Value": "YMBavekB2gEGAAAAMTEyMjU2AiQAAAAwODI0MTEyMi0wMTdiLWI0NzMtMDMxNi1jYmEwOTlkM2IwZWUAAAAAAAAAAAAAAAABAAAACQAAAFBEQV9CQVNFRAEAAAAAAAAAAAAAAAAAAAAAAAAAEgAAAEZDXzIwMjNfMDlfMDVfMDAxNAEAAAD6////AAAAAAYAAAAAAAAAAQAAAAAAAAAMAAAAWUdBR19QQkktcGJpBwAAAFVOS05PV04GAAAAKm51bGwqBgAAACpudWxsKgYAAAAqbnVsbCoGAAAAKm51bGwqBgAAACpudWxsKgYAAAAqbnVsbCpjAAAACgAAAF9MT1RfTk9ORV8AAAAAAAAAAAEAAADKembt6uEZwGrPzXBdFNw/fWr87QZ0G0AGAAAAKm51bGwq",
            "SourceTimestamp": "2023-07-09T05:10:33.5197086Z"
          },
          "ns=1;s=TransportUnitEnd": {
            "Value": {
              "Clock": "2023-10-18T17:37:21.15Z",
              "UserName": "112256",
              "TransportUnitEndSystemId": 1,
              "TuEndId": "*null*",
              "LotId": "_LOT_NONE_"
            },
            "SourceTimestamp": "2023-07-09T05:10:33.5277084Z"
          },
          "ns=1;s=DieHasBeenPicked": {
            "Value": {
              "Clock": "2023-10-18T17:37:01.26Z",
              "UserName": "112256",
              "DiePickMid": "*null*",
              "PickupVacuum": -1,
              "ComponentId": "27241122-017b-b4d6-0316-ef2e483eccd5",
              "EjectorLifetimeCounter": -1
            },
            "SourceTimestamp": "2023-07-09T05:10:33.5277084Z"
          },
          "ns=1;s=TransportUnitStart": {
            "Value": {
              "Clock": "2023-10-18T17:37:23.47Z",
              "UserName": "112256",
              "TransportUnitStartSystemId": 2,
              "TuStartId": "*null*",
              "LotId": "_LOT_NONE_"
            },
            "SourceTimestamp": "2023-07-09T05:10:33.5277084Z"
          },
          "ns=1;s=ProcessStateMachineChangedState": {
            "Value": {
              "Clock": "2023-10-18T17:37:25.25Z",
              "UserName": "112256",
              "ProcessState": 7
            },
            "SourceTimestamp": "2023-07-09T05:10:33.5277084Z"
          }
        }
      }
    ]
  },
  "enqueuedTime": "Wed Oct 18 2023 16:37:40 GMT+0200 (Central European Summer Time)",
  "properties": {
    "$$MessageSchema": "application/x-network-message-json-v1"
  },
  "systemProperties": {
    "iothub-connection-device-id": "XYZ",
    "iothub-connection-module-id": "msft-opc-publisher",
    "iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "638332222702179027",
    "iothub-enqueuedtime": 1697639860001,
    "iothub-message-source": "Telemetry",
    "x-opt-sequence-number": 5388332,
    "x-opt-offset": "5755283072872",
    "x-opt-enqueued-time": 1697639860157
  }
}

To Reproduce Steps to reproduce the behavior:

  1. Start the OPC Publisher with the command line settings --strict --mm=PubSub --me=Json.
  2. Have an OPC-UA Server that exposes a Variable Node that has a DataType of a multilevel Structure as indicated above. Note that in our case the multilevel Structure also has an optional Field on the inner and deepest level Structures, we don't know if this is the reason for the observed faulty behavior.
  3. Ensure that the OPC Publisher is subscribing to the Node indicated on Step 2.
  4. Publish the messages to an IoT Hub.
  5. Observe that the message contains the value of the multilevel Structure Node serialized in Binary instead of Json.

Expected behavior We expect the custom multilevel Structure to be serialized properly in Json and not in the Binary form.

Screenshots Here are some screen shots of the Information Model built with the Siemens SiOME Tool as well as the UaExpert while the OPC-UA Server is running.

Object with the exposed Variables (highlighted the Variable where we see the faulty serialization, all the others work as expected): image

Some of the inner custom DataTypes (Structures in bold) where the Structure optional Field is defined (DataTypes defined under Types>DataTypes>BaseDataType>Structure>CartesianCoordinates): image

See below the screenshots of another inner Structure (PbiBondType) as well as the root DataType Structure (PostBondingInspectionType) that the Variable PostBondingInspection shown on the server Objects (EVOMachine.EventsCollection) points to . See the DataTypes of the fields that point to the other Structures to create the multilevel Structure. See also the Target Node Id name (ns=1;s=PostBondingInspection.Binary) of the Forward ReferenceType HasEncoding with a Target Default Binary (the same naming convention was applied to all the other Structures defined on the Information Model): image

In UaExpert, here's the expansion of the DataTypeDefinition of the PostBondingInspectionType on the multiple screenshots below (note that the DefaultEncondingId.Identifier maps to the Forward ReferenceType Identifier indicated above, maybe this is relevant for your analysis): image image image image image

See also the screenshots for the inner Structures:

Here's the Object with the Variable and a sample value:

The OPC-UA Server was developed in house using the Traeger SDK, here's an excerpt of the data types definition:

[UPDATE TO ADD THE PUBLISHED METADATA] And here is the metadata message published by the OPC Publisher, which contains useful information for the bug analysis, because:

Desktop (please complete the following information):

Additional context The problem seems to be similar to the one reported here: #1334 that was closed without conclusion. @koepalex, @mregen we can provide the information model file (*.Nodeset2.xml) on a direct message if needed.

nelsonmorais commented 1 year ago

Updated the original description to include the OPC Publisher metadata message as it looks like it might contain relevant information for the bug analysis.

nelsonmorais commented 1 year ago

Hi @marcschier I've seen that you've labeled this issue to be addressed on v2.9.3, is there an ETA that you can share? Thanks!

marcschier commented 1 year ago

Hi @marcschier I've seen that you've labeled this issue to be addressed on v2.9.3, is there an ETA that you can share? Thanks!

Goal is to release 2.9.3 end of November, but since this bug must be addressed in the OPC UA stack dependency, we might need to push this out to 2.9.4 if we do not have an updated nuget before end of November.

nelsonmorais commented 1 year ago

@marcschier Thanks for the update!

Given that this issue might take some time to be fixed and this is blocking us to get access to an important set of data form one of our machines that we've just connected and for which we were not expecting this issue, is there any workaround that you could share that would allow us to solve the problem in the meantime, so that we can get access to the data?

If not, any "push" to have this issue addressed sooner would be greatly appreciated 😄

From our end, and if possible, we could even consider to temporarily do and use a private build from the source on this repository once the fix is available from your end and a final release (v2.9.3, v2.9.4) is not published yet.

marcschier commented 1 year ago

Once the fix is checked into the OPC-F repo, I will build a preview image that is using the preview nuget of the stack. using this preview build you could help us confirm that the issue is really addressed.

mregen commented 1 year ago

Hi @nelsonmorais, I think the issue is fixed in the stack PR because there are types derived from the CartesianCoordinates abstract type used for some structure members, which is the same issue. But to be sure it needs additional verification.

nelsonmorais commented 1 year ago

@mregen Thanks for the update! Let me know when and where can I get access to a preview build of the OPC Publisher that I can try to let you know if the problem is resolved at our end.

marcschier commented 1 year ago

Reopening to track the integration of the preview stack nuget.

marcschier commented 11 months ago

A latest build is available at mcr.microsoft.com/iotedge/opc-publisher:2.9.3-preview2. Closing, please re-open if issues persist.

nelsonmorais commented 11 months ago

Thanks @marcschier!