Not all system properties available while subscribing to Azure IoT Hub Events

[arkakk93]

Expected Behavior

Should be able to extract additional system properties for D2C IoT Hub messages from the delivered HTTP subscription message Example: message-id (Reference: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct#system-properties-of-d2c-iot-hub-messages)

Actual Behavior

Currently the only supported list of system properties is listed here: https://docs.dapr.io/reference/components-reference/supported-pubsub/setup-azure-eventhubs/#subscribing-to-azure-iot-hub-events

[CodeMonkeyLeet]

Seems like a reasonable extension of #227 to include user settable IoT Hub system properties.

[CodeMonkeyLeet]

The existing EventHubs bindings implementation exposes all the Event.SystemProperties extracted by the azure-event-hubs-go/v3 SDK, with the expectation that additional properties should be retrievable from the Annotations map but a cursory check using just az CLI simulated events don't generate any additional properties, so this will need custom events generated to test those fields specifically.

@arkakk93 Do you any details for how you're setting the device messages so we can look at replicating this?

[CodeMonkeyLeet]

Summarizing some investigation into SystemProperties of D2C IoT Hub messages not explicitly handled by Dapr EventHubs components:

• message-id
  • It looks like this property is mapped as an AMQP message property by IoT Hub, and the azure-event-hubs-go/v3 SDK already maps this back out to their Event.ID property.
  • Dapr EventHubs components can extract this and map it back to a message-id metadata property today.
  • If anyone has a pressing need specifically for just message-id we can break that apart as a one-off fix.
• user-id
  • This similarly looks like it gets mapped to AMQP message properties by IoT Hub, but is not handled by azure-event-hubs-go/v3 SDK today.
  • If the Go SDK moves in the direction of exposing the underlying AMQP message like the .NET SDK with EventData.GetRawAmqpMessage(), then Dapr can also extract this directly from the UserId and map it back to a user-id metadata property. Until then (or until azure-event-hubs-go surfaces the property another way), Dapr doesn't have visibility into this property.
• dt-subject and dt-dataschema
  • These are specific to Digitial twin events and dt-dataschema in particular is not user-settable, requiring a device twin environment with telemetry configured to test.
  • Requires further investigation at the azure-event-hubs-go/v3 SDK level for how these properties are received and could be exposed.

As a side note, it turns out that az iot device simulate can actually simulate the properties in question, for example:

az iot device simulate -n my-iothub -d my-test-device --msg-count 2 --msg-interval 1 --protocol http --properties "iothub-userid=my-user-id-value;iothub-messageid=my-message-id-value;dt-subject=my-digital-twin-id"

Enumerating some of the less obvious SystemProperties that can be passed to --properties:

• dt-subject
• iothub-contenttype (must be valid content type such as application/json)
• iothub-contentencoding (if iothub-contenttype defined as application/json, valid values: utf-8, utf-16, utf-32)
• iothub-correlationid
• iothub-messageid
• iothub-userid (value is base64-encoded in json because the underlying type is byte array)

[artursouza]

/cc @mukundansundar

[arkakk93]

The existing EventHubs bindings implementation exposes all the Event.SystemProperties extracted by the azure-event-hubs-go/v3 SDK, with the expectation that additional properties should be retrievable from the Annotations map but a cursory check using just az CLI simulated events don't generate any additional properties, so this will need custom events generated to test those fields specifically.

@arkakk93 Do you any details for how you're setting the device messages so we can look at replicating this?

I am directly relying on the messages generated by the Azure IoT Hub. These are routed to the Event Hubs and then subscribed too

[arkakk93]

Along with system properties, similar thing was observed for the Properties. When using Azure IoT Hub, the Enrichment details are sent to the Event Hub event as Properties. These are currently not available with Dapr

[arkakk93]

• message-id

@CodeMonkeyLeet, having the message-id will at least unblock us

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[CodeMonkeyLeet]

Removing stale tag and updating:

• message-id is now provided via #1221
• user-id could be provided using the azure-event-hubs-go v3.3.16 release
  • The v3.3.16 release depends on the updated go-amqp v0.16.0, which contains breaking changes
  • The azure-service-bus-go SDK used in the service bus pubsub and bindings components also depends on go-amqp, so it needs to be updated at the same time to v0.11.3 to avoid breaking the components-contrib build.
  • The v0.11.3 azure-service-bus-go causes the conformance tests to hang (deadlocks), which appears to be a regression in that SDK.
  • Dapr can't take this change until those issues are root caused and resolved.
    • @mukundansundar this can be spun off and tracked as a separate issue so that this issue can be closed, or you can repurpose this issue to track remaining progress on user-id support.
• dt-subject and dt-dataschema have no proposed path forward, so any support from them will come later and should be tracked by a separate issue.

[dapr-bot]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[dapr-bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

[uriel-kluk]

Is it possible to include dt-dataschema in all non-telemetry events as well?

We're in the process of automating our digital twin migration. Due to the unpredictable order of events, we often encounter telemetry or device twin change events. Creating a new digital twin with the correct ModelId becomes challenging when the model is not available in the event. Having dt-dataschema included in every event could simplify this process and help avoid race conditions.