arcus-azure / arcus.messaging

Messaging with Microsoft Azure in a breeze.
https://messaging.arcus-azure.net
MIT License
20 stars 11 forks source link

Support deferred messages or delayed processing #411

Open fgheysels opened 1 year ago

fgheysels commented 1 year ago

We might have processes where we receive a message from servicebus that cannot be processed yet. (For instance because we're waiting on another message coming from another party first). In such situations, it would be good if we could defer the message or resubmit the same message. It would be nice if we could do this in the implementation of the message-handler. I can see that the AzureServiceBusMessageHandler now already has functionality to Complete or Abandon messages via protected methdos. Some 'defer' or 'resubmit' logic there would be appropriate I think.

I think there are 2 approaches:

Re-submitting:

I'm thinking of having the possibility of re-submitting the exact same message to Service Bus again, but with a specific / specified ScheduledEnqueueTimeUtc property. Docs on this here. (That would probably mean completing the initial message and then resubmitting a copy of it , with all the same properties etc... but with a specified ScheduledEnqueueTimeUtc ? (Or can this be implemented by abandoning the message and passing in the necessary 'propertiesToModify' ? [ pass in ScheduledEnqueueTimeUtc as propertyToModify ? )

Deferring

This might be another possible approach, but I think it's more complex? Some documentation on the defer-functionality that is provided by Azure ServiceBus can be found here. The drawback of deferring -as I see it-, is that you need another mechanism to retrieve deferred messages again, and that you need to specify which deferred message you want to retrieve again (so you need to keep state / administration for this).

stijnmoreels commented 1 year ago

That would be a cool feature, indeed. I see some similarity with #25 and #36 .

fgheysels commented 1 year ago

That would be a cool feature, indeed. I see some similarity with #25 and #36 .

I see some similarity with #25 indeed. However, @tomkerkhove mentions there that the pump should orchestrate this, but I don't know how the pump could do that. The pump has no knowledge of the 'business logic' that must be executed (what must be executed first). Therefore, my assumption would be that the message-handler would be responsible for 'resubmitting' the message if it cannot be processed yet.

fgheysels commented 1 year ago

I've tried implementing this by calling the AbandonAsync method and specify a scheduledEnqueueTimeUtc property while abandoning, but this doesn't give us the desired results.

This issue is related to what we want to achieve here: https://github.com/Azure/azure-service-bus/issues/454

One of the workarounds that is proposed in the above issue is to 'Defer' the message and keep track of the deferred message by sending another message to the queue which contains the sequence-number of the deferred message so that it can be retrieved later. I think this might be a good approach for a work around.

I was thinking of having something like:

Of course, this is still -imho- a workaround for the problem at hand. Ideally, MSFT solves this by adding this functionality in ServiceBus as this work-around brings a few other problems to the table:

stijnmoreels commented 1 year ago

Sounds like a plan! šŸŽ‰ Thx a lot for documenting/investigating this so thoroughly. Will see if I can work on something while you're gone. šŸ‘

stijnmoreels commented 1 year ago

The feature is done being designed and almost picked up for development: https://github.com/Azure/azure-service-bus/issues/454#issuecomment-1645816895 Maybe we can wait a bit for it.

fgheysels commented 1 year ago

Well, I think this is something we're going to need very soon in a project. Also, other teams might find it very useful. I don't know how soon MSFT will start development, and how soon it will be released / available. I guess we can close that gap for now using our 'work around implementation', and then, once MSFT has provided a cleaner solution, modify our implementation to make use of the feature that MSFT has developed ?

stijnmoreels commented 1 year ago

Yes, think that is a good approach. Unless of course this is very critical for other projects, otherwise I would be in favor to wait a bit for Microsoft to catch up.

fgheysels commented 1 year ago

I think it is a crucial feature, something that should have been implemented by MSFT from the start imho. I'm afraid it can take up to 6 months or more before MSFT will deploy this.

Another thing regarding the possible implementation. I've discussed this with @gverstraete yesterday as well, and his first idea on how to implement this, could also be an option:

When 'deferring' a message, we could create a clone of the message, (with the same body and properties) and send this message to service bus with a `ScheduledEnqueuetimeUtc', and next to that we must then also make sure to 'accept' the original message so that it is no longer available in the queue.

stijnmoreels commented 1 year ago

Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.

Otherwise, this is indeed a rather elegant solution. If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.

fgheysels commented 1 year ago

Ah yes, but I'm wondering what Impact this could have on the performance/scaling/cost of Service Bus as for every message, there will be two in the queue.

Service Bus bills you per operation, not by the amount of messages. Service Bus Pricing. Of course, in a situation where you need to retrieve a deferred message, that 's an additional operation, but I would say that additional costs for this would be marginal ?

Otherwise, this is indeed a rather elegant solution. If we're thinking about implementing this, we should also probably note somewhere that the authorization of the Service Bus resource will require Write (new messages) too, instead of Read only.

That's a good point, and indeed a drawback.

fgheysels commented 1 year ago

The thing is that we'll need to decide on how to continue with this:

I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.

What's your take @gverstraete ? This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?

gverstraete commented 1 year ago

The thing is that we'll need to decide on how to continue with this:

  • implement using deferring the message and enqueue a message that says 'retrieve the deferred message'
  • implement via the approach that 'clones' the original message and sends it with a scheduled date, abandon the original message
  • wait for MSFT

I would be in favor to not to wait for MSFT as we don't know how soon the feature will be delivered. We can close the gap for now with an alternative approach. Once MSFT comes up with their solution, we can rework our workaround to use the 'official implementation'.

What's your take @gverstraete ? This is a feature we'll certainly require at a project I'm working on with @jcools85 right now, so he might have some ideas as well ?

Agree, I think cloning makes most sense

fgheysels commented 1 year ago

For implementation, I would go with 'cloning' the message. That looks to be the most simple solution.

Next to that, we must take into consideration that a processor not only needs to have listen permissions to the queue (or topic), but also needs to have send permission. In my opinion, that is something that we can add in the documentation. When a message is being deferred (send back to the queue), and the connection that we're using doesn't have send rights, we'll just throw an exception.

When MSFT has finished it's implementation, we can refactor our internal workings to make use of the functionality MSFT provides. If no 'send' rights are required for MSFT's implementation, we can modify our documentation and make a note of this in our release-notes.

jcools85 commented 2 months ago

Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.

stijnmoreels commented 2 months ago

Any update on this? We need this feature at NxtPort to implement a retry mechanism with custom backoff times.

We are working on a circuit breaker, but that would not include deferred messaging. See also this feature request on Microsoft's Service Bus, as there is some missing functionlity.

No time/budget to implement a custom solution as of yet. But we are always happy to receive PR's šŸ˜‰.

stijnmoreels commented 2 months ago

I think it is possible, though, to defer the message yourself in the message handler, add an application property with a timestamp, and filter with another message handler for certain past times. But as I've said, no room to take anything else at the moment.

fgheysels commented 2 weeks ago

I think it is very hard to implement this in the MessageHandler itself. First of all, Arcus currently doesn't expose a DeferMessage method or something similar. Next to that, the 'application developer' would need to be able to retrieve the deferred message (that wouldn't be that difficult), but the developer would need a way to send this message through the Arcus message-pump.

It looks like MSFT is not very responsible on my requests in this issue so my plan for a workaround in Arcus would be like this:

  1. Introduce a DeferMessageAsync(TimeSpan delay) method. This method would call the DeferMessageAsync method of the `ServiceBusReceiver
  2. Since we're now responsible for retrieving the deferred message ourselves; the DeferMessageAsync method of Arcus should send a scheduled message to servicebus which contains the details of the deferred message. (It definitely needs the SequenceNumber of the deferred message).
  3. An Arcus ServiceBus messagepump must have a default MessageHandler (or another mechanism) for receiving and processing the scheduled message that we've sent in step 2. This mechanism must make sure that it can interpret the message, retrieve the stored SequenceNumber and retrieve the deferred message via the SequenceNumber using the ReceiveDeferredMessage method of the ServiceBusReceiver.
  4. The Arcus MessagePump would then also need to make sure that the received deferred message is sent to the Arcus MessageRouter so that the correct registered message-handler can process the message.

More info on deferring in ServiceBus can be found here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-deferral#message-deferral-apis. The approach I mentionned here is actually more or less described in the 'sample scenarios' paragraph of that article.

stijnmoreels commented 2 weeks ago

We're talking about the same thing here šŸ˜‰

fgheysels commented 2 weeks ago

We cannot do that in the messagehandler. We currently have no access to the ServiceBusMessage itself, nor do we have a way to make sure that the deferred message that we would then retrieve, is send through the message-router.