Open arkakk93 opened 2 years ago
Seems like a reasonable extension of #227 to include user settable IoT Hub system properties.
The existing EventHubs bindings implementation exposes all the Event.SystemProperties
extracted by the azure-event-hubs-go/v3 SDK, with the expectation that additional properties should be retrievable from the Annotations
map but a cursory check using just az CLI simulated events don't generate any additional properties, so this will need custom events generated to test those fields specifically.
@arkakk93 Do you any details for how you're setting the device messages so we can look at replicating this?
Summarizing some investigation into SystemProperties of D2C IoT Hub messages not explicitly handled by Dapr EventHubs components:
message-id
Event.ID
property.message-id
metadata property today.message-id
we can break that apart as a one-off fix.user-id
user-id
metadata property. Until then (or until azure-event-hubs-go surfaces the property another way), Dapr doesn't have visibility into this property.dt-subject
and dt-dataschema
dt-dataschema
in particular is not user-settable, requiring a device twin environment with telemetry configured to test.As a side note, it turns out that az iot device simulate
can actually simulate the properties in question, for example:
az iot device simulate -n my-iothub -d my-test-device --msg-count 2 --msg-interval 1 --protocol http --properties "iothub-userid=my-user-id-value;iothub-messageid=my-message-id-value;dt-subject=my-digital-twin-id"
Enumerating some of the less obvious SystemProperties that can be passed to --properties
:
dt-subject
iothub-contenttype
(must be valid content type such as application/json)iothub-contentencoding
(if iothub-contenttype
defined as application/json, valid values: utf-8, utf-16, utf-32)iothub-correlationid
iothub-messageid
iothub-userid
(value is base64-encoded in json because the underlying type is byte array)/cc @mukundansundar
The existing EventHubs bindings implementation exposes all the
Event.SystemProperties
extracted by the azure-event-hubs-go/v3 SDK, with the expectation that additional properties should be retrievable from theAnnotations
map but a cursory check using just az CLI simulated events don't generate any additional properties, so this will need custom events generated to test those fields specifically.@arkakk93 Do you any details for how you're setting the device messages so we can look at replicating this?
I am directly relying on the messages generated by the Azure IoT Hub. These are routed to the Event Hubs and then subscribed too
Along with system properties, similar thing was observed for the Properties. When using Azure IoT Hub, the Enrichment details are sent to the Event Hub event as Properties. These are currently not available with Dapr
- message-id
@CodeMonkeyLeet, having the message-id will at least unblock us
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.
Removing stale
tag and updating:
message-id
is now provided via #1221 user-id
could be provided using the azure-event-hubs-go v3.3.16 release
user-id
support. dt-subject
and dt-dataschema
have no proposed path forward, so any support from them will come later and should be tracked by a separate issue.This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.
Is it possible to include dt-dataschema
in all non-telemetry events as well?
We're in the process of automating our digital twin migration. Due to the unpredictable order of events, we often encounter telemetry or device twin change events. Creating a new digital twin with the correct ModelId
becomes challenging when the model is not available in the event. Having dt-dataschema
included in every event could simplify this process and help avoid race conditions.
Expected Behavior
Should be able to extract additional system properties for D2C IoT Hub messages from the delivered HTTP subscription message Example: message-id (Reference: https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct#system-properties-of-d2c-iot-hub-messages)
Actual Behavior
Currently the only supported list of system properties is listed here: https://docs.dapr.io/reference/components-reference/supported-pubsub/setup-azure-eventhubs/#subscribing-to-azure-iot-hub-events