Azure / azure-event-hubs-node

Node client library for Azure Event Hubs https://azure.microsoft.com/services/event-hubs
MIT License
50 stars 46 forks source link

How to receive data from different partitions? #209

Closed makoit closed 5 years ago

makoit commented 5 years ago

Actually I receice my data from one partition, the partition 0. Now we have changes in our backend so I need to receive data from partition 0 and 1. I had a look to eventHubClient.d.ts and found this:

receive(partitionId: string | number, onMessage: OnMessage, onError: OnError, options?: ReceiveOptions): ReceiveHandler;

So I can only receive data from one partition? The parameter partitionId is a single value. Is there a option to have a array with multiple partitions for the receive method? If not how can I read from multiple partitions?

Thanks!

ShivangiReja commented 5 years ago

@makoit Yes, We can receive messages from multiple partitions using Event-processor-host.

It will also checkpoint metadata about the received messages at regular interval in an Azure Storage Blob. This makes it easy to continue receiving messages from where you left at a later time.

You can refer this sample. It focus on EPH (Event Processor Host) which is responsible for receiving messages. You can also use this example that sends multiple messages batched together.

makoit commented 5 years ago

@ShivangiReja thanks for your response. I will try it.

I have a service that receives data from the eventhub of partition 0. Then I noticed by the timestamp that data is missing. Then I noticed that data is also on partition 1. Now I want to receive and forward data from both partitions simultaneously to a third party system.

When I have to use the EventHubClient and when I have to use Event Processor Host ? Actually I use the EventHubClient.

Thanks!

ShivangiReja commented 5 years ago

@makoit When you want to receive events from a particular partition then you will use EventHubClient and when you want to receive events from multiple partitions then you will use Event Processor Host.

In your use case either you can create 2 EventHubClient(One for partition 0 and another for partition 1) or you can use Event Processor Host, It will create EventHub Receivers across all the partitions in the provided consumer group of an EventHub and provide you messages received across all the partitions.

General overview of how the Event Processor Host SDK works internally can be found over here.

ramya-rao-a commented 5 years ago

Closing this issue as the relevant information has already been shared above.