MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.2k stars 21.35k forks source link

Help needed with access to Microsoft.DigitalTwins.Twin.Update event routes #98160

Closed ogiel closed 1 year ago

ogiel commented 2 years ago

Hey Microsoft,

The page that I added this feedback to is unclear about how to access properties in a DigitalTwin update patch document. While using an event route on telemetry data, the properties can be accessed like this: $body.temperature > 22 However, it is unclear how to acces this data on twin update events. It clearly says that accessing the modelId differs and using $body.modelId works fine. But how can I access a property in the update patch document? I tried all sorts of combinations but nothing seems to work. As far as I can see the $body contains a key 'patch' which contains a list of updated properties.

So the first question is: How can I use event routes based on properties on twin update events (Microsoft.DigitalTwins.Twin.Update) instead of telemetry (microsoft.iot.telemetry) data?

And the second question: We currently do the Twin updates with a patch document directly from IoT Hub instead of sending Telemetry to the DigitalTwin and then updating the properties with a patch document. Should we be sending telemetry to the DT first and then updating twin properties? I see this done in the documentations but I do not really see added value (only added cost and complexity). However, if there is a good reason to do this then question #1 would be expired.

Thanks in advance,

Giel


Document Details

Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ogiel commented 2 years ago

Example for telemetry:

type = 'microsoft.iot.telemetry' AND
dataschema = 'dtmi:digitaltwins:rec_3_3:core:MultiSensor;1' AND
$body.temperature > 22

Example with twin update:

type = 'Microsoft.DigitalTwins.Twin.Update' AND
$body.modelId = 'dtmi:digitaltwins:rec_3_3:core:MultiSensor;1' AND
??? ($body.temperature does not work)
YashikaTyagii commented 2 years ago

@Fuchio Thanks for your feedback! We will investigate and update as appropriate.

SatishBoddu-MSFT commented 2 years ago

Hello @baanders Could you please help us in this matter.

ogiel commented 2 years ago

To update on this, we found that the docs have a note that says this functionality is not yet available for twin updates (filters on arrays to be precise).

However, the question on what the added value is of first pushing as telemetry data and then updating twin properties instead of updating twin properties right away remains.

baanders commented 2 years ago

@Fuchio Thank you for your question, and for the follow-up!

I'm working with the product team to get an answer to your second question and will report back shortly.

baanders commented 2 years ago

Update: The expert on this topic is out of office, returning next Monday. I will follow up here after conferring with them next week.

ogiel commented 1 year ago

Hey @baanders, has the expert returned to office?

I have one more thing to add as it is becoming a bit more confusing now. I wanted to setup a new simulation script that pushes telemetry and then updates the twin properties through another function with an eventgrid trigger by ADT and stumbled upon this documentation: https://learn.microsoft.com/en-us/azure/digital-twins/how-to-ingest-iot-hub-data.

These docs are about sending telemetry from IoT Hub to a digital twin and this uses the same logic that we currently have. However, that means they also skip the PublishTelemetryAsync() function making it impossible to trigger event routes on thresholds.

This might mean that we have got the idea wrong on where we should trigger events (alerts). Should we use a script that checks all twin updates if thresholds are met instead of finding a way to trigger the function only when the threshold is met?

baanders commented 1 year ago

Hi @Fuchio,

We're still talking about it, but I'm here to check in with a few points!

  1. To update twin properties according to IoT Hub telemetry data, you've found the right document; you can update the properties with a JSON patch document as described in the Ingest telemetry from IoT Hub article.

  2. Are you looking at a particular document that's showing a different process with an additional step of sending telemetry--either through the PublishTelemetryAsync() function you mentioned in your latest comment, or perhaps the SendTelemetry API? These functions shouldn't be required to update a twin, so we want to make sure we update the docs to be clearer about that if needed.

  3. For your latest question-- Since filtering on arrays isn't currently available, unfortunately yes, you'd need to parse all change notifications if you want to read the actual value in the array (like in your example of triggering if a value is over some threshold). One option that may make this a bit easier is to use filters on other properties outside of the array to help reduce the amount of parsing, like filtering on the subject (twin ID) or modelId.

Hopefully some of this additional info is helpful!

ogiel commented 1 year ago

Hey @baanders,

Thanks for the follow up, I really appreciate the effort.

  1. We do currently use it like the docs you linked, so just updating digital twin properties directly after IoT Hub.
  2. I am not necessarily looking for an example with the PublishTelemetryAsync(), but I mostly wondered if that was the supposed way to go since it would enable the event route filtering on actual data (like temperature values).
  3. Alright, that makes it clear (and for us that means we have implemented it in the right way for now). We currently have an azure function that runs everytime a twin of modelId gets updated. Then in the Azure Function we loop over the array and check if (for examle) the path is "/temperature". However, that means we have to run the AZ Function on every sensor instead of just the ones with values over a certain threshold while still triggering the filter an equal amount of times.

The thing that I am still confused about is what the purpose of the Telemetry flow in ADT is. Currently the benefits of PublishTelemetryAsync would be that it would make it possible to trigger event routes on values in the body. But the examples do not use it and it would also add an AZ Function trigger for every incoming message.

Thanks for the information, hopefully in the future we can do filters like body.temperature in the event routes :D. Really appreciate the effort from everyone in this thread to get to these answers.

baanders commented 1 year ago

Hi @Fuchio,

  1. Yep, that workflow is correct. To look within the patch, you'll need to look into each change notification.

As you mentioned, PublishTelemetryAsync does require the initial patch parsing, as well as an additional function app trigger and round trip to the twin. It's not included in the main suggested flow for ingesting telemetry into ADT from IoT Hub, so for your scenario it sounds like updating the twin with a patch document is working well.

baanders commented 1 year ago

I've made a few updates to the article as a result of this thread.

  1. Copied over the note that filtering on array fields isn't supported, so that it also appears in the section on digital twin change notifications. Hopefully that will make this information more visible.
  2. Updated some misleading wording in the telemetry messages section that previously indicated these telemetry notifications were received by digital twins. This section actually describes the telemetry that's emitted by digital twins when the SendTelemetry API is used. Hopefully that will help avoid another area of confusion around telemetry flow.

These changes are being made in this pull request: PR #211893, Update event descriptions

Following the conversation on this thread and the updates to the document, I'm going to mark this issue as resolved once the pull request is merged. Feel free to continue commenting in the thread as needed, as we can always reopen the conversation if necessary.

Thank you!