Ability to purge a queue

[jtaubensee]

From @SeanFeldman on December 20, 2016 5:25

There are scenarios where purging a queue is necessary

• Testing scenarios
• Removing all messages because they are no longer needed
• Clearing out deferred messages when sequence numbers are no longer retained by an app

Similar functionality: ASQ`s CloudQueue.ClearAsync().

This would apply to both queue/subscription and associated DLQs.

Copied from original issue: Azure/azure-service-bus-dotnet#46

[jtaubensee]

This is something we can revisit in the future. We currently have an internal project that would make this much easier for us. The project will manage state against partitions.

[jtaubensee]

From @SeanFeldman on January 27, 2017 0:42

We currently have an internal project that would make this much easier for us.

Where's the OSS community in "us" when it's internal? Sharing is carrying, right? 😃

[jtaubensee]

The project I am referring to is for our cluster management. It will simplify transactions across shards, allowing us to better ensure that messages have been purged from all.

[SeanFeldman]

Is this something that would be considered? Thank you.

[binzywu]

@SeanFeldman it's on our list. But it may take sometime for us to reach to that item. 😢

[SeanFeldman]

@binzywu knowing it's planned is better than just wonder if it's even considered 🙂 Thank you.

[SeanFeldman]

Just pinging for an update 🙂 Thank you.

[ddobric]

Any news on this?

[SeanFeldman]

Still wish this functionality would exists...

[tidusjar]

Still waiting for this!

[lansal15]

Two years and counting since this was opened, has there been any progress made?

[ben-burton]

I am trying to clean up some acceptance tests by deleting a topic and recreating it after each test fixture. In theory this should isolate the tests but I'm experiencing timing issues asking Azure to delete and create an entity in quick succession. I tried adding some logic using the Exists method on the Microsoft.Azure.ServiceBus.Management.ManagementClient - which would seem to indicate things are in a clean state, but this didn't work; possibly because things haven't fully materialised in Azure yet?

Maybe a less heavy handed approach and something like Clear or Purge would help me here. I am in need of a way to clean topics/queues/DLQs between tests

[erickbp]

Hi, is there an update on this? thanks

[yahorsi]

Please consider implementing, needed in our case e.g. in the integration tests where we need to make sure the queue is empty before every new test runs.

Thank you

[SeanFeldman]

@yahorsi if that helps, I do a "test run ID" header. Any message received with a different value that is set for the current test run is ignored. That way while there's no way to purge, your tests will be purging and running. Not perfect, but perhaps it would help in your case. Cheers.

[SeanFeldman]

Not a month that someone doesn't ask how to purge a queue... If Queue Storage can do it, Service Bus should be able as well.

/cc @axisc

[codeputer]

I'm asking for a purge feature - during dev messages get messed up, and they have to be cleared before testing can begin again. Everybody writes their own utility to receive and dump the messages in the queue . Why is this such an issue to implement?

[old-profile]

Please consider implementing this feature. Thank you

[lansal15]

+1

[vinilneves]

+1

[gitgandhi]

+1

[JGroot]

+1 This feature would be extremely helpful

[macalbert]

+1

[tgado]

+1

[DavidBogues]

+1

[gabrielmsdiniz]

+1

[sorinslavic]

plus one

[EldertGrootenboer]

Hi, this has surfaced to the top of our backlog, and we hope to implement this in the coming months.

[SeanFeldman]

this has surfaced to the top of our backlog, and we hope to implement this in the coming months.

I'm getting emotional reading this after so many years. Thank you.

[priyankar-dutta]

Any idea when this might go live?

[EldertGrootenboer]

Any idea when this might go live?

Not yet, when we have more information I will update this issue.

[payiAzure]

any ETA on when this feature could be live?

[diligent176]

Following... Service Bus Explorer (preview) in Azure portal is missing some critical features. Most notably, there is no easy way to purge the dead letter queue. And when re-sending messages from the dead letter queue, there is no option to remove the message. This means you need to use "Receive and Delete" mode blindly, perhaps unintentionally removing a new dead letter that just arrived in the queue - rather than the one(s) you intended to receive and delete.

[Trubador]

You can purge a queue in Service Bus like this:

Go to the queue => service bus explorer (preview)

Then do the below as the first thing to purge the queue, remember to change from peek mode to receive mode in the dropdown:

it might also work with dead letter queue.

[SeanFeldman]

@diligent176 @Trubador, this issue concerns service (broker) exposing a feature, not a way to achieve a purge.

[Trubador]

@diligent176 @Trubador, this issue concerns service (broker) exposing a feature, not a way to achieve a purge.

I don't understand, please elaborate.

[diligent176]

@SeanFeldman My fault... I led the conversation in the wrong direction.
Please ignore my comment about SB Explorer preview. This open issue is not the same.

@Trubador To clarify my comment, yes I know the web version can "Receive and Delete" but it does so blindly. It is a destructive receive for all the messages. What I really want in the portal is to select one message from the DLQ, replay it and delete it. There is no "safe" way to delete a single message of your choosing from the DLQ.

[SeanFeldman]

@diligent176, no worries.

@Trubador, this repository is for Service Bus broker bugs and feature requests, not tools or SDKs. Those have separate repositories.

The issue was raised as a feature request to be able to purge queues as a service side exposed functionality, w/o the need to receive and delete, looping through messages in the entity. This is what all tools (SBExplorer or the portal) are doing today. Hope that helps to clarify my comment.

[Trubador]

@SeanFeldman My fault... I led the conversation in the wrong direction. Please ignore my comment about SB Explorer preview. This open issue is not the same.

@Trubador To clarify my comment, yes I know the web version can "Receive and Delete" but it does so blindly. It is a destructive receive for all the messages. What I really want in the portal is to select one message from the DLQ, replay it and delete it. There is no "safe" way to delete a single message of your choosing from the DLQ.

If you can find the sequence number or message id for the message and assign a lock to it you should be able to delete it: https://learn.microsoft.com/en-us/rest/api/servicebus/delete-message DELETE http{s}://{serviceNamespace}.servicebus.windows.net/{queuePath}/messages/{messageId|sequenceNumber}/{lockToken}

[Trubador]

@diligent176, no worries.

@Trubador, this repository is for Service Bus broker bugs and feature requests, not tools or SDKs. Those have separate repositories.

The issue was raised as a feature request to be able to purge queues as a service side exposed functionality, w/o the need to receive and delete, looping through messages in the entity. This is what all tools (SBExplorer or the portal) are doing today. Hope that helps to clarify my comment.

I gotcha.

[SeanFeldman]

Me thinking about this feature as I go through piles and piles of messages in the testing environment: https://www.youtube.com/watch?v=84Tq-eAJIk4 No, I cannot delete the entity.

[SeanFeldman]

@EldertGrootenboer, another soul that'd hug you if this feature were there.

[jchannon]

@EldertGrootenboer howdy! i'd like to delete all messages in a queue via the REST API. Let's go! 😀

[vanhoutenbos]

You can purge a queue in Service Bus like this:

Go to the queue => service bus explorer (preview)

Then do the below as the first thing to purge the queue, remember to change from peek mode to receive mode in the dropdown:

it might also work with dead letter queue.

Thats all fun and games untill you want to clear 500k messages. It takes about 2 minutes to ReceiveAndDelete 500 messages, not planning on clicking that button a 1000 times :P other suggestions?

Still +1 for queue purge feature like in RabbitMQ

[grujicd]

You can purge a queue without 1000 clicks on a button from QueueExplorer (disclaimer: it's commercial and I'm the author). But behind the scene it just does Receive so it will take a while to purge 500k messages. It's sad that ancient MSMQ has much better API for some things. Not only purge, but you could delete a message from the middle of the queue without having to receive (and abandon) all messages in front.

[EldertGrootenboer]

We are currently working on a feature that will allow batch-delete operations, which will be the first step to a full purge functionality. This batch-delete feature is in active development and release is planned in the first half of 2024.

[SeanFeldman]

@EldertGrootenboer, is there a spec to look at before the work get wrapped up? My specific concern is about batch size and semantics around it.

PS: is Binzy really working on this?! 😂

[max-favilli]

You can purge a queue in Service Bus like this:

Go to the queue => service bus explorer (preview)

Then do the below as the first thing to purge the queue, remember to change from peek mode to receive mode in the dropdown:

it might also work with dead letter queue.

In the screenshot you are on the queue, if you move to the dead letters can you still set the number of messages to receive? I am trying but it seems limited to 1 max.

[SeanFeldman]

@max-favilli this issue is for service-side purging. Questions about tools such as Portal Explorer or SBExplorer are better answered on StackOverflow.

[max-favilli]

@max-favilli this issue is for service-side purging. Questions about tools such as Portal Explorer or SBExplorer are better answered on StackOverflow.

Is there any such limitation service wide?

Because even programmatically with receivemessagesasync (https://learn.microsoft.com/en-us/dotnet/api/azure.messaging.servicebus.servicebusreceiver.receivemessagesasync?view=azure-dotnet) no matter how high you set maxMessages and maxWaitTime, if you are receiving from deadletter subqueue, you only get 1 message at time.