[EventProcessorHost] Using consumerGroup option not honoring old lease or making new

[jzwack]

Describe the bug When running first without consumerGroup option, storage container is created and leases are managed (using $Default of course). However if you go back, add consumerGroup option, it does not use the old container if you use "$Default". If you use a new consumerGroup name, it does not create a new folder in the blob.

To Reproduce See above

Expected behavior Using consumerGroup = "$Default" should use the existing folder/lease. Using a new valid consumer group name (e.g., "local") should create a new folder in the container.

Package-name: azure-event-processor-host Package-version: @azure/event-processor-host@1.0.2 node.js version: 8.9.3 OS name and version: Azure Windows App Service

Additional context I want to also make sure this is correct usage of the option:

const eph = EventProcessorHost.createFromConnectionString(
      EventProcessorHost.createHostName(eventHubHostName),
      storageConnectionString,
      leaseContainerName,
      ehconnectionString,
        {
            eventHubPath: entityPath,
            leaseDuration: 60,
            leaseRenewInterval: 30,
            // consumerGroup: consumerGroup,
            onEphError: (error) => {
                logger.info(">>>>>> [%s] Error: %0", eventHubHostName, error);
            }
        }
    );

[amarzavery]

@jzwack - How are you creating the leaseContainerName? Are you using something like this:

const leaseContainerName = EventProcessorHost.createHostName("test-container");

If yes, then this will always create a new storage container, since that will append a guid string to the provided prefix test-container. You may just want to provide const leaseContainerName = "someUniqueName"; and keep on using the same name between successive runs or multiple EPH instances.

• One more thing, the default value for consumerGroup is "$default" -> d is lowercase. Just want to be sure that you provided the same value second time and not "$Default".

[jzwack]

Yes for instance on local I'm using: const leaseContainerName = EventProcessorHost.createHostName("eph-local-container");

For dev (the Azure Webapp) I'm using: const leaseContainerName = EventProcessorHost.createHostName("eph-dev-container");

Now both of these I deployed without specifying a consumer group, and you're right it created a folder in the container called "$default" (no uppercase). I'll try again with it correct. Strange in Azure Portal and sbexplorer they show "$Default" as the name. Oh well.

However when I came back and specified in the options:: consumerGroup: "local"

No new folder was created in the container, it connected fine, but never received events.

[amarzavery]

However when I came back and specified in the options: consumerGroup: "local" No new folder was created in the container, it connected fine, but never received events.

Yup, it is by design. One storage container will only have one folder in it (and it represents that consumer group). If you want to listen from a different consumergroup then you need to create a different storage container.

There are multiple reasons for doing this:

1. A consumer group is a logical bucket that groups receivers together. EventHub supports creating epoch receivers where the receiver with the highest epoch value is only allowed to receive messages from a given partition within a consumer group. This ensures that only 1 receiver is receiving messages at a time. This helps us implement the competing consumer pattern. You wouldn't want multiple receivers receiving from the same partition at the same time, as that will lead to duplicate messages.
2. To make EPH function faster, we try to make less http requests to the blob container. This is done by making 1 call to the storage container to list all the blobs. If we have multiple folders in the container, it will give all the blobs across all the folders. Now it will be extremely difficult to manage duplicate receivers across all the partitions.

May I ask, why do you need to receive messages from different consumer groups at the same time?

[ramya-rao-a]

Closing as by design.