Open fgheysels opened 1 year ago
That would be a cool feature, indeed. I see some similarity with #25 and #36 .
That would be a cool feature, indeed. I see some similarity with #25 and #36 .
I see some similarity with #25 indeed. However, @tomkerkhove mentions there that the pump should orchestrate this, but I don't know how the pump could do that. The pump has no knowledge of the 'business logic' that must be executed (what must be executed first). Therefore, my assumption would be that the message-handler would be responsible for 'resubmitting' the message if it cannot be processed yet.
I've tried implementing this by calling the AbandonAsync
method and specify a scheduledEnqueueTimeUtc
property while abandoning, but this doesn't give us the desired results.
This issue is related to what we want to achieve here: https://github.com/Azure/azure-service-bus/issues/454
One of the workarounds that is proposed in the above issue is to 'Defer' the message and keep track of the deferred message by sending another message to the queue which contains the sequence-number of the deferred message so that it can be retrieved later. I think this might be a good approach for a work around.
I was thinking of having something like:
a (protected) method in AzureServiceBusMessageHandler
that you can call to 'Defer' or re-submit a message using a method that has a signature like this: DeferMessageAsync( DateTimeOffset resubmitDateTime);
.
When this method is called it should:
ProcessMessageEventArgs.DeferMessageAsync
method.scheduledEnqueueTimeUtc
property of this new message must be set to the resubmitDateTime
value that was passed as a parameter.When the pump / messagerouter receives the message that has been posted because of the deferral, the messagerouter should be able to retrieve the original deffered message via the ReceiveDeferredMessageAsync
method and the sequenceNumber
that was stored in that message. Once the deferred message is retrieved, it should be routed to the correct MessageHandler, just as we're doing now with 'normal' messages.
(I believe for this to work, we would need to be able to register a message-handler that can handle those messages that have been posted because of a deferral ? Maybe this can be done by wrapping the necessary information in a DeferredMessageMetadata
type or something ? When such a type is received, the messagehandler that is linked to this type can be executed. The implementation of this message-handler retrieves the deferred message, and makes sure that the messaage is routed ? Just thinking out loud).
Of course, this is still -imho- a workaround for the problem at hand. Ideally, MSFT solves this by adding this functionality in ServiceBus as this work-around brings a few other problems to the table:
Sounds like a plan! š Thx a lot for documenting/investigating this so thoroughly. Will see if I can work on something while you're gone. š
The feature is done being designed and almost picked up for development: https://github.com/Azure/azure-service-bus/issues/454#issuecomment-1645816895 Maybe we can wait a bit for it.
Well, I think this is something we're going to need very soon in a project. Also, other teams might find it very useful. I don't know how soon MSFT will start development, and how soon it will be released / available. I guess we can close that gap for now using our 'work around implementation', and then, once MSFT has provided a cleaner solution, modify our implementation to make use of the feature that MSFT has developed ?
Yes, think that is a good approach. Unless of course this is very critical for other projects, otherwise I would be in favor to wait a bit for Microsoft to catch up.
I think it is a crucial feature, something that should have been implemented by MSFT from the start imho. I'm afraid it can take up to 6 months or more before MSFT will deploy this.
Another thing regarding the possible implementation. I've discussed this with @gverstraete yesterday as well, and his first idea on how to implement this, could also be an option:
When 'deferring' a message, we could create a clone of the message, (with the same body and properties) and send this message to service bus with a `ScheduledEnqueuetimeUtc', and next to that we must then also make sure to 'accept' the original message so that it is no longer available in the queue.
Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.
Otherwise, this is indeed a rather elegant solution. If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.
Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.
Service Bus bills you per operation, not by the amount of messages. Service Bus Pricing. Of course, in a situation where you need to retrieve a deferred message, that 's an additional operation, but I would say that additional costs for this would be marginal ?
Otherwise, this is indeed a rather elegant solution. If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.
That's a good point, and indeed a drawback.
The thing is that we'll need to decide on how to continue with this:
I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.
What's your take @gverstraete ? This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?
The thing is that we'll need to decide on how to continue with this:
- implement using deferring the message and enqueue a message that says 'retrieve the deferred message'
- implement via the approach that 'clones' the original message and sends it with a scheduled date, abandon the original message
- wait for MSFT
I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.
What's your take @gverstraete ? This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?
Agree, I think cloning makes most sense
For implementation, I would go with 'cloning' the message. That looks to be the most simple solution.
Next to that, we must take into consideration that a processor not only needs to have listen permissions to the queue (or topic), but also needs to have send permission. In my opinion, that is something that we can add in the documentation. When a message is being deferred (send back to the queue), and the connection that we're using doesn't have send rights, we'll just throw an exception.
When MSFT has finished it's implementation, we can refactor our internal workings to make use of the functionality MSFT provides. If no 'send' rights are required for MSFT's implementation, we can modify our documentation and make a note of this in our release-notes.
Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.
Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.
We are working on a circuit breaker, but that would not include deferred messaging. See also this feature request on Microsoft's Service Bus, as there is some missing functionlity.
No time/budget to implement a custom solution as of yet. But we are always happy to receive PR's š.
I think it is possible, though, to defer the message yourself in the message handler, add an application property with a timestamp, and filter with another message handler for certain past times. But as I've said, no room to take anything else at the moment.
I think it is very hard to implement this in the MessageHandler itself. First of all, Arcus currently doesn't expose a DeferMessage
method or something similar. Next to that, the 'application developer' would need to be able to retrieve the deferred message (that wouldn't be that difficult), but the developer would need a way to send this message through the Arcus message-pump.
It looks like MSFT is not very responsible on my requests in this issue so my plan for a workaround in Arcus would be like this:
DeferMessageAsync(TimeSpan delay)
method. This method would call the DeferMessageAsync
method of the `ServiceBusReceiverDeferMessageAsync
method of Arcus should send a scheduled message to servicebus which contains the details of the deferred message. (It definitely needs the SequenceNumber
of the deferred message).MessageHandler
(or another mechanism) for receiving and processing the scheduled message that we've sent in step 2. This mechanism must make sure that it can interpret the
message, retrieve the stored SequenceNumber
and retrieve the deferred message via the SequenceNumber
using the ReceiveDeferredMessage
method of the ServiceBusReceiver
.MessageRouter
so that the correct registered message-handler can process the message.More info on deferring in ServiceBus can be found here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-deferral#message-deferral-apis. The approach I mentionned here is actually more or less described in the 'sample scenarios' paragraph of that article.
We're talking about the same thing here š
We cannot do that in the messagehandler.
We currently have no access to the ServiceBusMessage
itself, nor do we have a way to make sure that the deferred message that we would then retrieve, is send through the message-router.
We might have processes where we receive a message from servicebus that cannot be processed yet. (For instance because we're waiting on another message coming from another party first). In such situations, it would be good if we could defer the message or resubmit the same message. It would be nice if we could do this in the implementation of the message-handler. I can see that the
AzureServiceBusMessageHandler
now already has functionality to Complete or Abandon messages via protected methdos. Some 'defer' or 'resubmit' logic there would be appropriate I think.I think there are 2 approaches:
Re-submitting:
I'm thinking of having the possibility of re-submitting the exact same message to Service Bus again, but with a specific / specified
ScheduledEnqueueTimeUtc
property. Docs on this here. (That would probably mean completing the initial message and then resubmitting a copy of it , with all the same properties etc... but with a specified ScheduledEnqueueTimeUtc ? (Or can this be implemented by abandoning the message and passing in the necessary 'propertiesToModify' ? [ pass inScheduledEnqueueTimeUtc
as propertyToModify ? )Deferring
This might be another possible approach, but I think it's more complex? Some documentation on the defer-functionality that is provided by Azure ServiceBus can be found here. The drawback of deferring -as I see it-, is that you need another mechanism to retrieve deferred messages again, and that you need to specify which deferred message you want to retrieve again (so you need to keep state / administration for this).