search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.14k stars 4.91k forks source link

Support partitionID in Azure eventhub input #36907

Open Martin-Kemp opened 11 months ago

Martin-Kemp commented 11 months ago

Describe the enhancement:

Currently partitionID is not supported in Azure eventhub input. It's commented out in the code because Azure SDK apparently doesn't support it. See commented out code here: https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/azureeventhub/input.go#L197 and here: https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/azureeventhub/eph.go#L77

Is this still the case?

As far as I can see partitionID is supported: https://github.com/Azure/azure-event-hubs-go

Describe a specific use case for the enhancement or feature:

PartitionID is very useful for downstream applications consuming eventhub data, we use MQTT topic strings in our partitionID and use this to route data.

elasticmachine commented 11 months ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Martin-Kemp commented 10 months ago

I can confirm that the new SDK works. I tested this basic example: https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/messaging/azeventhubs/example_consuming_events_test.go and followed this documentation: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs#section-readme

I sent a message to an eventhub and specified the partitionKey then read the message and got the partitionKey:

Received message: Your message content, PartitionKey: %!d(string=your-partition-key), sequence number: 0, enqueued time: 2023-10-27 10:34:41.948 +0000 UTC

By the way, it turns out I was looking for PartitionKey, not PartitionID. But both should be available with the new SDK.