AutoDelete subscriptions on idle - No way to configure SubscriptionProperties

[sfmskywalker]

The AzureServiceBusTransport class implements ISubscriptionStorage that is then responsible for creating and managing Azure topic subscriptions, which is awesome.

However, I'm running into the following issue:

• My app runs in a multi-node cluster (Azure App Service with horizontal auto-scaling enabled)
• As new containers get created, new subscriptions are created for a given topic.
• As containers get destroyed, these subscriptions remain on Azure Service Bus.
• These "phantom" subscriptions keep receiving messages sent to the associated topic, eventually reaching the storage limit for the topic.

Some more background

For this multi-node setup, I am using the container's HOST name to prefix the input queue when creating an IBus. The rationale being: if I want pub/sub, my containers need a unique queue name to prevent other containers from "stealing" messages from the queue (which would happen if they all consume the same queue).

Some pictures about the topology and the issue:

So I have one topic and 7 subscriptions (there were 7 containers running at some point)

Here are the 7 subscriptions:

The first subscription I want to be auto-removed, because I removed that container (my laptop) from the cluster. Because if I don't, the queue keeps receiving messages:

But since the container is gone, there is no consumer to process this queue.

Possible solutions

Barring the potential fact that I might have set things up completely wrong and stupid, here's what I am thinking to auto-delete orphaned subscriptions (and corresponding queues):

1. Use SetAutoDeleteOnIdle on the transport. This works for the queue, but the subscription remains active, which then causes a new queue to be created. So I figured, maybe I can just derive my own version of AzureServiceBusTransport and configure the subscription it creates internally with AutoDeleteOnIdle like this: subscription.AutoDeleteOnIdle = TimeSpan.FromMinutes(5).
2. Keep track of the created subscriptions in some database and have some background worker clean them up (inspired from this SO post).
3. Someone tells me I'm doing it all wrong and I need to take a different route 😄

Option 1 seems to be the simplest, but as it stands, I would have to duplicate a lot of code from the Rebus.AzureServiceBus package.

But, if you agree, we can make a few small changes in the AzureServiceBusTransport class like creating a protected virtual method called e.g. CreateSubscription that returns the created subscription, allowing the sub class to further configure the subscription (like setting the SetAutoDeleteOnIdle property.

Another option might be to assume the SetAutoDeleteOnIdle setting should be applied not only to created queues, but subscriptions as well. But that may be too coarse-grained for other users.

[mookid8000]

Another option might be to assume the SetAutoDeleteOnIdle setting should be applied not only to created queues, but subscriptions as well. But that may be too coarse-grained for other users.

This is 110% the correct way to handle this!

Except! 😐 If I have understood things correctly, subscriptions are considered idle if they haven't had anything sent/received/peeked for a while, but they will have messages arrive as long as events are being published to their topic, so I'm afraid that a subscription will never be considered idle if the topic is active.

So... I am unsure whether there's a nice solution here. IMO Azure Service Bus is modeled completely backwards with the way subscriptions are aggregated beneath topics and not beneath queues.

The fact that a subscription can be configured to forward messages to a queue, which can then be (auto-)deleted is just evidence that it's wrong. But it also enforces by suspicion that the auto-delete feature has been bolted onto Azure Service Bus, probably just because RabbitMQ has it, without thinking possible scenarios through.

I am not sure I can think of a nice way to handle this. Maybe you're better off writing a simple background worker, which will traverse your Azure Service Bus topics and their subscriptions from time to time, to see if any subscriptions are configured to forward messages to non-existing queues and then delete them?

[sfmskywalker]

Thanks for confirming!

Yeah good point about what conditions will cause a subscription to become idle. I was hoping I might have misunderstood, but you're probably right. I might do some experiments to make sure, but we'll see. For now I'm explicitly unsubscribing during (graceful) application shutdown, which is what seems to happen most of the times.

The fact that a subscription can be configured to forward messages to a queue, which can then be (auto-)deleted is just evidence that it's wrong. But it also enforces by suspicion that the auto-delete feature has been bolted onto Azure Service Bus, probably just because RabbitMQ has it, without thinking possible scenarios through.

I'm glad you're saying this; I was thinking, like, surely these guys know what they're doing and I'm just missing something! 😬

Depending on how things currently look with the orphaned subscriptions becoming a problem or not, I might indeed write a background worker to clean up the mess.

Thanks for your thoughts on all this, I appreciate it!