search

Azure / azure-event-hubs-go

Golang client library for Azure Event Hubs https://azure.microsoft.com/services/event-hubs
MIT License
88 stars 69 forks source link

Sending data to a specific partitionId #274

Closed MartinKosicky closed 1 year ago

MartinKosicky commented 1 year ago

Hello I have a question, because I don't understand here how am I supposed to send events on a specific partition (via partition ID not partitionKey). I want to make a fan out of some event and I dont understand how am I supposed to do it with the API. I know I can make a Hub instance for every partition but making 16 connections seems a little strange.

C# has this feature https://learn.microsoft.com/en-us/dotnet/api/azure.messaging.eventhubs.producer.sendeventoptions?view=azure-dotnet

MartinKosicky commented 1 year ago

Note: I'll try this:

event.SystemProperties.Annotations["x-opt-partition-id"] = somePartitionId
richardpark-msft commented 1 year ago

Hi @MartinKosicky, sending to a partition is something you configure at the link level (i,e when you create the hub). You can use the HubWithPartitionedSender function when creating your eventhub.

hub, err := eventhub.NewHubFromConnectionString(cs, eventhub.HubWithPartitionedSender("partition-id"))
MartinKosicky commented 1 year ago

Yes that is why i said it is insane to make a new connection just to send sometimes to the partition. I dont get why its not possible on the session level. I guess ill use the rest api for this.

richardpark-msft commented 1 year ago

Yes that is why i said it is insane to make a new connection just to send sometimes to the partition. I dont get why its not possible on the session level. I guess ill use the rest api for this.

Sorry about that, I totally missed that sentence.

The design of this particular package really does emphasize using partition keys and isn't built around swapping between links for different senders. So it has the exact problem you mentioned.

Rather than using the REST API, if you're willing to try out our new beta library, is to use our new beta package: github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs, which supports what you want to do right out of the box with ProducerClient.

https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/messaging/azeventhubs/example_producing_events_test.go#L52

The PartitionID is just an option you specify when you create your batch but it's all maintained transparently and will only use a single TCP connection (with multiple links, one per partition).

richardpark-msft commented 1 year ago

(leaving open in case you have any other questions)

richardpark-msft commented 1 year ago

Closing, as there doesn't appear to be any further discussion for this.