romankiss commented 2 years ago

First off all, thanks for this feature known as a VETER (Validate-Enrich-Transform-Enrich-Route) pattern.

The reason why I am sending this feedback is, that all documents (included also the REST APIs) are written without any respect to the device model based on the DTDL version 2.

The device model (represented by its modelId) is the common part between the IoT ends such as the device and IoT Central App (IoT Hub), but for REST APIs and in the new feature such as a Data Transformation are used an internally instance of the device template and all queries are sourced based on that. From the device model point of the view (such as the device and app sides) we can have multiple shareable interfaces, components, etc. across multiple different device templates.

So, my question is, how the query or/and data transformation can be done in the respect of the shareable device model interfaces, components, etc. I have tried to make a very simple transformation query to export a simple property such as the modelId, but with no result. Note, that the query for device template is used an undocumented name such as a capabilityModelId, which it is a hardcoded name for modelId (this indicates no consistencies between the device model and device template)

Thanks Roman

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f02dd3c7-73e6-5178-b2b2-6b849a63ae3c
Version Independent ID: c82392d3-4ac6-7df6-5652-54071e48b0a7
Content: Transform data inside Azure IoT Central
Content Source: articles/iot-central/core/howto-transform-data-internally.md
Service: iot-central
GitHub Login: @dominicbetts
Microsoft Alias: dobett

SatishBoddu-MSFT commented 2 years ago

Hello @romankiss Thanks for the feedback.

Our SME will review and get back on this.

dominicbetts commented 2 years ago

Hi @romankiss - thanks for the detailed feedback. Can I make sure that I've understood the key points correctly. I'll focus on the transformation feature in data export for now, rather than the REST API. Ill also use the "Temperature Controller" multiple component sample for reference:

I believe you're asking about accessing information about the device template and its device model in the transformation queries?

The raw data that's available to the transformation looks like (some details removed for brevity):

{
  "applicationId": "450b9a64-8ea3-417d-8540-a7b2288c292b",
  "enrichments": {},
  "messageSource": "telemetry",
  "telemetry": [
    {
      "id": "dtmi:com:example:Thermostat:__temperature;1",
      "name": "temperature",
      "value": 49.51243191047629
    }
  ],
  "enqueuedTime": "2021-12-02T14:17:02.382Z",
  "messageProperties": {},
  "device": {
    "types": [],
    "templateName": "Temperature Controller",
    "properties": {
      "reported": [
        {
          "id": "dtmi:com:example:Thermostat:__maxTempSinceLastReboot;1",
          "name": "maxTempSinceLastReboot",
          "value": 86.87871839450773,
          "component": "thermostat1"
        },
        ...
      ]
    },
    "blocked": false,
    "templateId": "urn:modelDefinition:irbkwhz46hi:xypyc9osr5",
    "approved": true,
    "provisioned": true,
    "name": "Temperature controller",
    "simulated": false,
    "modules": [],
    "id": "tc-01",
    "cloudProperties": [],
    "organizations": []
  },
  "component": "thermostat1"
}

The templateId field contains the system assigned ID for the device template.
The DTML modelID dtmi:com:example:TemperatureController;2 isn't in the raw data.
The DTDL component that sent the data is identified by name in the component field.

It is possible to reference the component name and use it in a transformation. For example:

import "iotc" as iotc;
if .component then
    {
        deviceId: .device.id,
        component: .component,
        timestamp: .enqueuedTime,
        temperature: .telemetry | iotc::find(.name == "temperature").value
    }
else
    empty
end

If your model uses interface inheritance, you have no access to the interface IDs in the transformation. For example, this model:

{
    "@id": "dtmi:com:example:TemperatureController;2",
    "@type": "Interface",
    "contents": [ … ],
    "displayName": {
        "en": "Temperature Controller"
    },
    "extends": [
        "dtmi:myContosoQuickstartApp:Battery;1"
    ],
    "@context": [
        "dtmi:iotcentral:context;2",
        "dtmi:dtdl:context;2"
    ]
}

{
    "@id": "dtmi:myContosoQuickstartApp:Battery;1",
    "@type": "Interface",
    "contents": [
        {
            "@id": "dtmi:myContosoQuickstartApp:Battery:Level;1",
            "@type": "Telemetry",
            "displayName": {
                "en": "Level"
            },
            "name": "Level",
            "schema": "integer"
        }
    ],
    "displayName": {
        "en": "Interface"
    },
    "@context": [
        "dtmi:iotcentral:context;2",
        "dtmi:dtdl:context;2"
    ]
}

Generates raw data that looks like this for the level telemetry:

{
  "applicationId": "450b9a64-8ea3-417d-8540-a7b2288c292b",
  "messageSource": "telemetry",
  "messageProperties": {},
  "enrichments": {},
  "telemetry": [
    {
      "name": "level",
      "value": 80
    }
  ],
  "enqueuedTime": "2021-12-02T14:17:03.523Z",
  "device": {
    "types": [],
    "organizations": [],
    "provisioned": true,
    "modules": [],
    "id": "tc-01",
    "name": "Temperature controller",
    "templateId": "urn:modelDefinition:irbkwhz46hi:xypyc9osr5",
    "templateName": "Temperature Controller",
    "properties": {
      "reported": [  …  ]
    },
    "approved": true,
    "cloudProperties": [],
    "simulated": false,
    "blocked": false
  }
}

@Sekhar-Kutikuppala - do you have any comments to add at this point?

Thanks, Dominic

romankiss commented 2 years ago

Hi @dominicbetts, Thank you for your details. I understood, that the data transformation is depended from the input raw data and there is no problem if we have some internal data in the transformation like is a templateId, etc. (for instance: urn:modelDefinition:tadwaslrkxh:jo5kwjqgjp7, dtmi:com:example:TemperatureController:__serialNumber;2, ...)

Note, that exporting those internal data is useless because the IoT PnP ends are related to the device model described by the DTDL spec v2, which it is represented by its unique modelId property. This modelId property is well-known by IoT Central App as a common resource between the IoT PnP ends, therefore it should be as a part of the system properties available anywhere in the IoT Central App included also in the data transformation.

Currently, we have to use a REST call to obtain a modelId property based on the exported templateId.

Once again, the data transformation is a great feature and help in the integration pipeline with other services directly, for instance ingesting a telemetry data to the IoT Hub and/or IoT Central App, firing a property changes to the Event Grid, etc.

The data transformation is the great part of the VETER pattern and I hope that also the destination (R part of the VETER) will be possible to handle dynamically, for instance using an expression in the header value and url string (like we have it in the Event Grid).

Thanks Roman

dominicbetts commented 2 years ago

Hi @romankiss,

It sounds like you have two key requests:

The DTDL model ID is included in the raw data and available to a transformation query.
Dynamic destination handling.

If that's correct, can I suggest you add them to the feedback portal: https://feedback.azure.com/d365community/forum/35b59d83-f424-ec11-b6e6-000d3a4f0da0# -I notice you've already used it :).

I don't think any documentation update is needed right now? If you agree, I'll go ahead and close this issue.

Regards, Dominic

romankiss commented 2 years ago

Hi @dominicbetts,

that's correct. The following example shows firing an event to the Event Grid topic with CloudEventSchema:

Data transformation is a simple query:

{
    applicationId: .applicationId,
    deviceId: .device.id,
    modelId: "??",
    modelName: .device.templateName,
    messageType: .messageSource,
    enqueuedTime: .enqueuedTime,
    messageProperties: .messageProperties,
    enrichments: .enrichments,
    component: .component
}

The Webhook destination:

Event message received by AEG subscriber:

{
  "id":"123456",
  "source":"abc",
  "specversion":"1.0",
  "type":"dataexport",
  "data":{
    "modelId":"??",
    "deviceId":"device2000",
    "applicationId":"f6b559e3-1001-4a35-8209-090562865253",
    "enrichments":{},
    "properties":[
      {
        "value":38.88,
        "module":null,
        "component":"thermostat2",
        "name":"maxTempSinceLastReboot"
      }
    ],
    "enqueuedTime":"2021-12-06T12:37:56.835Z",
    "messageType":"devicePropertyReportedChange",
    "modelName":"Temperature Controller"
  }
 }

As you can see, the webhook endpoint headers are hardcoded, so in this case such as a cloud event message it can be OK as a part of the configuration, but in the case of ingestion a modified telemetry data back to the IoT Central App, we need something like the following:

url: https://{.hostmame}.azure-devices.net/devices/{.device.id}/messages/events?api-version=2021-04-12

headers: dt-subject: {.component} iothub-creation-time-utc: {.enqueuedTime}

So, I am going to write this needs as a feedback for next consideration features.

btw. It will be nice to have a built-in a self destination endpoint.

Thanks Roman

dominicbetts commented 2 years ago

Thanks!

please-close

romankiss commented 2 years ago

Hi @dominicbetts,

I am facing a runtime problem with an enqueuedTime property (it is a struct type) in the text like is shown in the following example:

As you can see the above screen snippet, there is no issue during the design time (the property is shown as a string type), but during the runtime there is an error:

Please, could you look at that issue.

Thanks Roman

romankiss commented 2 years ago

Hi @dominicbetts, did you have a chance to look at the issue around the enqueuedTime property described in my previously comment ?

Thanks Roman

dominicbetts commented 2 years ago

Hi @romankiss - You say that your enqueuedTime property is a struct type - can you show me an example of what it looks like in your input message? Thanks!

romankiss commented 2 years ago

Hi @dominicbetts, thanks for your reply. The enqueuedTime property is created by IoT Central, it's not my property.

The following screen snippet shows an input message for IoT Plug and Play mobile model, but it will be a part of every input message:

Thanks Roman

dominicbetts commented 2 years ago

Hi @romankiss - thanks for replying. I'm not sure what's happening here. When I try out your transformation with some sample data it works fine. Here's my transformation code that looks identical to yours:

{
    typeOfEnqueuedTime: (.enqueuedTime | type),
    msg: "XXX \(.enqueuedTime | tostring) YYY"
}

I'm sending to a webhook destination - here's the result:

dominicbetts commented 2 years ago

@Sekhar-Kutikuppala - can you help out with this question?

romankiss commented 2 years ago

Hi @dominicbetts, I have just created a new instance of the IoT Central App and it looks like it is working well.

Also, I can see, the enqueuedTime property is now a string type, not a struct type.

Thank you so much. Roman

dominicbetts commented 2 years ago

@romankiss - glad to hear it's all working as expected now.

MicrosoftDocs / azure-docs

Device model vs Device template #84661

Document Details

please-close