jsquire commented 2 months ago

Issue Transfer

This issue has been transferred from the Azure SDK for .NET repository, #45908.

Please be aware that @Babbel404 is the author of the original issue and include them for any questions or replies.

Details

Request

Currently the operation to defer a message is a one way operation. As a Service Bus Entity Manager I would like the ability to programmatically restore a message state to Active from Deferred.

Use Case

Scenario

As a Service Bus Entity manager I need the ability to manage queues and topic/subscriptions with fine-grain control over what messages I operate on. In my event driven system there are a large number of events coming from an external source. Sometimes there are values contained within these messages that cause issues. Under this scenario these messages are dead-lettered. It can be assumed that there exists some property about the messages that I can use to identify which messages were dead-lettered for a given reason. Sometimes this could be the DeadLetterReason other times this could be the EnqueuedTimeUtc, the actual property used to determine which messages I want to requeue or delete is not important.

Solution / Current Problem

I have successfully created a management application that can subscribe to Dead Letter sub queues and perform management actions on my behalf. This application can operate on any number of messages in the queue and perform actions based on filter criteria provided.

Actions:

Preserve
Delete
Requeue

However, I cannot fully implement the Preserve functionality. Ideally, messages would retain their current position on the Dead Letter queue. To my knowledge, this cannot be done without introducing an external system to store the messages. This has forced me to either Requeue or Delete these messages in order to find messages that match my search criteria on the queue.

I have created a version of this application that can use message deferral to set messages aside in the dead letter queue that do not match the search criteria. However, after this operation is done, I cannot restore the state to Active for these messages. This is problematic as it leaves me with limited options for these messages:

Requeue (and delete) the messages to the main queue
Delete the messages
Requeue the messages with a very short time to live to force a dead letter. This action modifies the DeadLetterReason and is not a good approach

Because the Dead Letter queue is a sub queue I cannot requeue the dead letter messages directly to the sub queue. So this is also not an option. Even if this were an option, it would be less desirable, as it would modify the message sequence order.

Example

Here is a contrived example that can help illustrate how this feature would help. I ask that the use case be evaluated at a more general level than just Dead Letter messages, as being able to utilize deferred messages in a normal queue may also make sense for some workflows.

Queue: MetrologyResultProcessZProcessor Info: Apply the logic for process Z to all messages from the MetrologyResult stream Volume: 10,000 - 100, 000 messages per hour

A change occurred in the measurement process that produces the metrology result for specific parts. This change caused several similar but unrelated failures in the code for a few part types. Part types C-145, R-872 and E-21 are known at this time to be impacted, however there could be other failures. There are currently 500,000 messages in the dead letter queue from this event, and potentially other unrelated failures.

The development team has published a fix for the MetrologyResultProcessZProcessor code that should address the issues in C-145 and R-872 and they have requested a small subset of the data be reprocessed from the dead letter queue, 100 messages each. After processing and validating the results for these initial messages, all dead letter messages for C-145 and R-872 should be requeued.

I would like to iterate over the dead letter messages and cherry pick out matching messages to reprocess. I want to perform this action in a durable way, and I would prefer if messages that did not match were left un-altered

Additional Notes

I feel that a simple code modification in the MessageReceiver similar to here we could expose a method to RestoreActive or RemoveDeferral. This new method simply needs to expose a way to interact with the AMQP Receiver DisposeMessageRequestResponseAsync and change the ManagementConstants.Properties.DispositionStatus

jsquire commented 2 months ago

@EldertGrootenboer : To my knowledge, this is not an operation exposed by the Service Bus contract. If I'm mistaken, would you please reopen the linked SDK repo issue and help us understand the protocol details? Thanks!

EldertGrootenboer commented 2 months ago

Basically we can distinguish two asks here.

The first ask is to allow to receive a specific message from the DLQ. This would mean that instead of the need to defer messages, it would just be possible to receive the specific messages that need to be resubmitted from the DLQ. For this ask, we would like to request to upvote and comment on this item in Azure Ideas.

The second ask is to allow moving a deferred message back to the original entity. This would allow to reprocess a deferred message using the default logic, instead of having to do this in the client that retrieves the deferred message. For this ask, we would like to request to upvote and comment on this item in Azure Ideas.

We will close this issue, as these are valid requests, however we want to consolidate these under the open Azure Ideas items that have been linked. This will help us better track the interest in these items, and accordingly provide this with the right priorities. We recommend both upvoting the item, as well as commenting on them with the scenario which the requested feature will help solve.

SeanFeldman commented 2 months ago

I honestly with this split brain between GitHub issue tracker and Azure Ideas would stop. Either using one or keeping them in sync. But having two is so counter and unnecessary.

@EldertGrootenboer, thoughts?

EldertGrootenboer commented 2 months ago

I honestly with this split brain between GitHub issue tracker and Azure Ideas would stop. Either using one or keeping them in sync. But having two is so counter and unnecessary.

@EldertGrootenboer, thoughts?

This is a valid concern that we have discussed internally as well. The idea that we have is to use Azure Ideas for any feature requests, and use GitHub for general technical issues. Customer specific technical issues will continue to go through Microsoft support. We also want to look into consolidating these where possible, to better keep track of specific asks, as now we might have the same requests in 2 or more locations.

That said, any issue that is already open here, and has had traction on it, we will probably keep open, even if this is a feature request, so we do not lose history and insights. For new feature requests, we will divert these to Azure Ideas, as has been done with this issue.

Babbel404 commented 2 months ago

Basically we can distinguish two asks here.

The first ask is to allow to receive a specific message from the DLQ. This would mean that instead of the need to defer messages, it would just be possible to receive the specific messages that need to be resubmitted from the DLQ. For this ask, we would like to request to upvote and comment on this item in Azure Ideas.

The second ask is to allow moving a deferred message back to the original entity. This would allow to reprocess a deferred message using the default logic, instead of having to do this in the client that retrieves the deferred message. For this ask, we would like to request to upvote and comment on this item in Azure Ideas.

We will close this issue, as these are valid requests, however we want to consolidate these under the open Azure Ideas items that have been linked. This will help us better track the interest in these items, and accordingly provide this with the right priorities. We recommend both upvoting the item, as well as commenting on them with the scenario which the requested feature will help solve.

Excellent, thank you for the feedback and instructions. I will respond to these linked Azure Idea items as instructed.

I would just like to air out a concern that have with the Azure Idea items linked. Given how old the tickets are and the relatively low traffic on the issues (one being created two years ago with no upvotes). I feel like I will be adding to a ticket that will never see the light of day.

I feel that the second link you provided for the deferred messages highlights exactly what I need. I feel that this functionality is something a lot of teams would want, however, its buried deep in the Azure Idea world and its difficult to find. Before creating this GitHub issue I did search online for any existing tickets or threads for this topic, however, I came up empty handed. This supports my thought that even if teams are looking for this feature request they will not find it.