MQTT Injest Paylod

[neildes2]

What is the MQTT json payload supposed to look like to map to the device file? I tried variations of the following with no luck.

{ "dataItemId": "avail", "value": "AVAILABLE" }

I see the payload come into the agent. It logs it. But no mapping.

Agent v2.2.0.13

[wsobel]

We don’t have a prescribed JSON format for Agent ingress. We support a SHDR payload, but if the payload begins with a curly brace “{“ we don’t have a way to interpret since there is no standard we are using.

See: https://mtcup.org/Pipeline_Architecture#mqtt-adapter-pipeline

The JsonMapper is currently disabled and JSON is handled via a custom transform for now.

See: https://mtcup.org/Data-Transformation-Using-Ruby#mqtt-pipeline-fix-execution-state-of-a-device (JsonMapper example)

If someone suggests a JSON payload that we can map into the MTConnect model, we could implement it. The original issue was the number of IIoT sensor data representations was too diverse to “guess” the payload, so we punted on the first version.

Suggestions are welcome.

On Nov 6, 2023, at 06:49, neildes2 @.***> wrote:

What is the MQTT json payload supposed to look like to map to the device file? I tried variations of the following with no luck.

{ "dataItemId": "avail", "value": "AVAILABLE" }

I see the payload come into the agent. It logs it. But no mapping.

Agent v2.2.0.13 https://user-images.githubusercontent.com/113067605/280748615-9e3c4668-6a2a-48d6-9fc8-d4298a7c658a.png https://user-images.githubusercontent.com/113067605/280748762-e29bbba4-dd1d-4509-8507-a917f189626a.png — Reply to this email directly, view it on GitHub https://github.com/mtconnect/cppagent/issues/365, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ6BCLM7TG5XTJTAYWA5V3YDD2IFAVCNFSM6AAAAAA67UKOBWVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TSMZVGEZTGNQ. You are receiving this because you are subscribed to this thread.

[robot-ranger]

as a first shot suggestion, mapping SHDR as closely as possible would result in something like this:

Basic Events; simple key:value pairs:

{
    "timestamp": 123456789,
    "dataItemId1": "value1",
    "dataItemId2": "value2",
    ...,
    ...,
    "dataItemIdN": "valueN"
}

Conditions simply require all 6 fields:

{
    "timestamp": 123456789,
    "conditionId": [
        "HIGH",
        "nativeCode",
        "nativeSeverity",
        "qualifier",
        "msgContent"
    ]
}

Messages:

{
    "timeStamp": 123456789,
    "messageId": [
        "nativeCode",
        "msgContent"
    ]
}

[robot-ranger]

another approach to conditions and messages is a dict. better human readability at the cost of some more programatic formatting and arguably extraneous characters for the computer machines:

Conditions w/ a dict:

{
    "timestamp": 123456789,
    "conditionId": {
        "level": "HIGH",
        "nativeCode": "nativeCode",
        "nativeSeverity": "nativeSeverity",
        "qualifier": "qualifier",
        "messge": "message",
    }
}

same could be applied to messages

[wsobel]

Another option is to have two forms:

• A simple key/value pair with id and value
• An object with named fields for things that are more complex.
  • Condition
  • Message
  • DataSets, Timeseries, and Tables.

[
  {
   "timestamp": "2023-11-06T12:12:44Z",
    "temp123": 22.6,
    "xpos": 1002.345,
    "exec": "ACTIVE"
  },
  {
   "timestamp": "2023-11-06T12:12:44Z",
    "dataItemId": "messageId",
    "nativeCode": "xxxxx",
    "value": "this is a message"
  },
  {
   "timestamp": "2023-11-06T12:12:44Z",
    "dataItemId": "conditionId",
    "conditionId": "XXX123",
    "nativeSeverity": "1000",
    "qualifier": "HIGH",
    "nativeCode": "ABC",
    "value": "something went wrong"
  },
  {
   "timestamp": "2023-11-06T12:12:44Z",
    "dataItemId": "avail",
    "value": "AVAILABLE"
  },
  {
   "timestamp": "2023-11-06T12:12:44Z",
    "dataItemId": "someTable",
    "value": {
      "A": {
        "cell1": 123,
        "cell2": 456
      },
      "B": {
        "cell1": 555,
        "cell2": 666        
      }      
    }
  }
]

[wsobel]

Or a somple degenerate form:

{
   "timestamp": "2023-11-06T12:12:44Z",
    "temp123": 22.6,
    "xpos": 1002.345,
    "exec": "ACTIVE",
   "cond": {
      "level": "fault",
      "conditionId": "XXX123",
      "nativeSeverity": "1000",
      "qualifier": "HIGH",
      "nativeCode": "ABC",
      "value": "something went wrong"
  }
}

[wsobel]

another approach to conditions and messages is a dict. better human readability at the cost of some more programatic formatting and arguably extraneous characters for the computer machines:

Conditions w/ a dict:

{
  "timestamp": 123456789,
  "conditionId": {
      "level": "HIGH",
      "nativeCode": "nativeCode",
      "nativeSeverity": "nativeSeverity",
      "qualifier": "qualifier",
      "messge": "message",
  }
}

same could be applied to messages

I'm gravitating towards something similar to this as well.

[wsobel]

@neildes2 What do you think of the latest suggestion from @robot-ranger ?

[robot-ranger]

im a little torn. i like using the dataItemId as the key for simple event data items with one field ('value'),

but i also like explicit keys like "level" and "nativeCode" for the more complex types.

I figured the field names like level, native code, etc didnt appear in SHDR, so thats why i suggested a list in the conditions and messages.

but im sort of splitting hairs. these all seem sensible to me.

[wsobel]

I think this from will give both:

{
   "timestamp": "2023-11-06T12:12:44Z",
    "temp123": 22.6,
    "xpos": 1002.345,
    "exec": "ACTIVE",
   "cond": {
      "level": "fault",
      "conditionId": "XXX123",
      "nativeSeverity": "1000",
      "qualifier": "HIGH",
      "nativeCode": "ABC",
      "message": "something went wrong"
  }
}

[robot-ranger]

in keeping with data item id as the key, should the key for the condition simply be the condition's data item id?

{
   "timestamp": "2023-11-06T12:12:44Z",
    "temp123": 22.6,
    "xpos": 1002.345,
    "exec": "ACTIVE",
    "conditonId": {
      "level": "fault",

      "nativeSeverity": "1000",
      "qualifier": "HIGH",
      "nativeCode": "ABC",
      "message": "something went wrong"
  }
}

[neildes2]

How about...

[ { "timestamp": "2022-10-21T17:20:41.4291Z"dataItemId": "avail", "value": "AVAILABLE" }, { "timestamp": "2022-10-21T17:20:41.4291Z"dataItemId": "execution", "value": "ACTIVE" }, { "timestamp": "2022-10-21T17:20:41.4291Z"dataItemId": "estop", "value": "ARMED" }, { "timestamp": "2022-10-21T17:20:41.4291Z"dataItemId": "mode", "value": "AUTOMATIC" } ]

And for conditions

[neildes2]

what about DATASETs and TABLES? Dictionary format?

[wsobel]

Data Sets and Tables will be key/value pairs–so either format could work. The value is a JSON object.

The question is, should we have one timestamp for all or individual timestamps?

[robot-ranger]

i think its 1x timestamp per payload. just like shdr. what is the use case of bundling various timestamps in one payload?

[robot-ranger]

the problem with

{
"timestamp": "2022-10-21T17:20:41.4291Z",
"dataItemId": "avail",
"value": "AVAILABLE"
}

is that you cant have multiple data items per payload without being careful; relying on alternating id & value. What happens if "value":"[some data]" is missing?

using the data items' own id as the key would enable multiple events in one payload:

{
"timestamp": 123456789,
"exe": "ACTIVE",
"prog": "1234.nc",
 "..":"..",
 "..":"..",
"estop": "ARMED"
}

[neildes2]

I agree with you. I like the id as the key. your suggestion below I like. And, ONE timestamp with the payload of multiple tags like shdr.

{
   "timestamp": "2023-11-06T12:12:44Z",
    "temp123": 22.6,
    "xpos": 1002.345,
    "exec": "ACTIVE",
    "conditonId": {
      "level": "fault",

      "nativeSeverity": "1000",
      "qualifier": "HIGH",
      "nativeCode": "ABC",
      "message": "something went wrong"
  }
}

[robot-ranger]

ok, so lets see a full example:

{
  "timestamp": "2023-11-06T12:12:44Z",
  // basic events:
  "temp123": 22.6,
  "xpos": 1002.345,
  "exec": "ACTIVE",
//conditions:
  "myConditonId": {
    "level": "fault",
    "nativeSeverity": "1000",
    "qualifier": "HIGH",
    "nativeCode": "ABC",
    "message": "something went wrong"
  },
//messages:
  "thisIsAMessageId": {
    "nativeCode": "ABC",
    "message": "this is a message"
  },
//table:
  "someTableId": {
    "A": {
      "cell1": 123,
      "cell2": 456
    },
    "B": {
      "cell1": 555,
      "cell2": 666        
    }    
  },
//data set:
  "aDataSet": {
    "v1": 123,
    "v2": 456,
    "v3": 789
  },
//time series:
  "aTimeSeriesId": {
    "count": 10,
    "frequency": 100,
    "values": [1,2,3,4,5,6,7,8,9,10]
  }
}

did i miss anything?

[wsobel]

That’s where I’m going. Traveling this week, will see how much I can get done. Got the basics finished using rapidjson, will push branch soon.

• W

On Nov 13, 2023, at 20:38, robot-ranger @.***> wrote:

ok, so lets see a full example:

{ "timestamp": "2023-11-06T12:12:44Z", // basic events: "temp123": 22.6, "xpos": 1002.345, "exec": "ACTIVE", //conditions: "myConditonId": { "level": "fault", "nativeSeverity": "1000", "qualifier": "HIGH", "nativeCode": "ABC", "message": "something went wrong" }, //messages: "thisIsAMessageId": { "nativeCode": "ABC", "message": "this is a message" }, //table: "someTableId": { "A": { "cell1": 123, "cell2": 456 }, "B": { "cell1": 555, "cell2": 666
}
}, //data set: "aDataSet": { "v1": 123, "v2": 456, "v3": 789 }, //time series: "aTimeSeriesId": { "count": 10, "frequency": 100, "values": [1,2,3,4,5,6,7,8,9,10] } } did i miss anything?

— Reply to this email directly, view it on GitHub https://github.com/mtconnect/cppagent/issues/365#issuecomment-1809402574, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ6BCMOK73XVJK72HQ6OLTYELDRZAVCNFSM6AAAAAA67UKOBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBZGQYDENJXGQ. You are receiving this because you commented.

[wsobel]

Check out the latest release, 2.2.0.15, for the implementation. I will add some more extensive error handling once we have agreed on this implementation.