Properties and System properties missing when subscribing to Azure Events pubsub

[ralarcon]

Expected Behavior

Using the PubSub or Input binding with Azure Event Hubs, I get all the relevant information: event body, properties and system properties.

Actual Behavior

The binding is missing important events data that are stored in properties and system properties. Other services send events to Event Hubs and use that properties and system properties to provide relevant information, for example, IoT Hub, where fixes to the binding has been done to include that important information:

There has been other issues related to this which have fixed the integration with Azure IoT Hubs and its compatible Event Hub endpoint:

https://github.com/dapr/components-contrib/issues/1151 https://github.com/dapr/components-contrib/issues/1243

In the case of Azure Digital Twins (ADT), when you route output data to Event Hubs, the following information is part of the event: Payload / Event Body:

{
    "modelId": "dtmi:contosocom:DigitalTwins:Thermostat;1",
    "patch": [
        {
            "value": 68.96053795403296,
            "path": "/Temperature",
            "op": "replace"
        }
    ]
}

Properties:

    "cloudEvents:id": "4b5798eb-bbfc-47c9-b51d-7caa1d21c1c1",
    "cloudEvents:source": "dt-101.api.weu.digitaltwins.azure.net",
    "cloudEvents:specversion": "1.0",
    "cloudEvents:type": "Microsoft.DigitalTwins.Twin.Update",
    "cloudEvents:time": "2023-02-10T07:11:37.2157194Z",
    "cloudEvents:subject": "thermostat1",
    "cloudEvents:traceparent": "00-f6adffd202e09341d346d52e9ff7d6ce-cf6d7f32fe23bb9e-01",
    "CorrelationId": "8038e0bb-15e6-4470-9d50-bc7bf6888080",
    "ContentType": "application/json"

System Properties:

    "x-opt-sequence-number": 2874,
    "x-opt-offset": 1838336,
    "x-opt-enqueued-time": "2023-02-10T07:11:37.923+00:00"

Here you can see the whole event data from a C# debug screenshot:

Steps to Reproduce the Problem

Use PubSub / Input Binding components with Azure Event Hubs + ADT output as described in this article: https://learn.microsoft.com/en-us/azure/digital-twins/concepts-route-events

Create a client application that use the DAPR eventhubs pub/sub (go or C#) and the properties and system properties are not present when receiving the data.

In the code there are some references to the "requireAllProperties" but not found a way to make it work: https://github.com/dapr/components-contrib/blob/f0587ca7488432ef6bddaecbb00dd4f01e1cc4cf/pubsub/azure/eventhubs/eventhubs.go#L130

Another way to test it is to create a client app that uses the PubSub with EventHubs, publis some events adding properties and validate that the properties or system properties are not present when subscribed to EventsHub.

[berndverst]

@ItalyPaleAle do you remember whether the IOT Hub properties are now being sent correctly after you switched the EventHubs implementation?

[grzegorzmusial-grundfos]

Pub/sub components was the only reason I started looking into DAPR few days ago. But this issue is blocking me for using it at all.

I'm using data enrichment in IotHub on custom endpoints. When using DAPR with Eventhub pub/sub, all custom message properties are skipped.

I found this piece of code, which is responsible for passing through properties from incoming EventHub message: https://github.com/dapr/components-contrib/blob/main/common/component/azure/eventhubs/events.go

Only short list of properties are supported, all others go to default "// nop" statement.

I can't figure out what could be the reason for not including them all? Or maybe metadata "requireAllProperties" should be renamed to "requireAlmostAllProperties" ;)

[AiHaibara]

hope to using dapr for scale out instance process eventhub message, cause direct using eventhub sdk seems will limit by partition count. (using dapr can helpful for this correct?)

will the all properties will support?

[AiHaibara]

hope to using dapr for scale out instance process eventhub message, cause direct using eventhub sdk seems will limit by partition count. (using dapr can helpful for this correct?)

it seems it will also limit by partition count, only one app will receive the request if partition count is 1.