search

rebus-org / Rebus.AzureServiceBus

:bus: Azure Service Bus transport for Rebus
https://mookid.dk/category/rebus
Other
33 stars 20 forks source link

Peek lock not renewed #72

Closed Tsjunne closed 2 years ago

Tsjunne commented 3 years ago

Hello,

We are experiencing some issues with duplicate message deliveries and i suspect that the peek lock is not getting renewed. We are using the .AutomaticallyRenewPeekLock() setting on a service that needs to handle pretty long processing times.

If I check the codebase I should be seeing something in the logs along the lines of "Renewing peek lock for message with ID {messageId}", but I don't see any logs like this. According to the code, the renewal will not happen with PreFetching enabled and I checked, we are not using it.

Our queues get created by Rebus (Rebus.AzureServiceBus 8.1.5) with a lock duration of 1 minute, so I would expect the renewal code to execute every 30s. Is it possible that the TotalMinutes get rounded down somewhere, so it becomes zero in case of the 1 minute default with time skew? So the interval is effectively disabled?

mookid8000 commented 3 years ago

Hi @Tsjunne , I would have expected the message's lease to have been renewed too in your case.

I've made some small adjustments around lease renewal, because I just realized that exceptions were swallowed if renewal failed – could you update to Rebus.AzureService 9.0.0-b2 and see if it makes a difference?

(BTW please note that the transport in v9 has been updated to use Microsoft's newest driver – it's fully compatible with Rebus.AzureServiceBus 8, it's just breaking in the sense that it has a different driver dependency)

Tsjunne commented 3 years ago

Hello, I saw that I was watching the 9.0.0 code and not the 8.1.5. Looks like a duplicate of #54, where we don't really see the underlying issue due to Exception swallowing. We'll await stabilization of the 9.0.0 release, since this uses the new MS libs and I see nothing potentially wrong with the 8.1.5 version of the Rebus code. I assume it's a Azure issue. We do see a lot of dependency failures on the Renew operations, but we are kind of blind without a proper logging.

maulik-modi commented 3 years ago

@Tsjunne or @mookid8000 , as 9.X is out, can you confirm if your issue is resolved?

mookid8000 commented 3 years ago

I haven't seen any problem with the new 9 release in this area, but my experience with it is also fairly limited.

Tsjunne commented 3 years ago

We have a pretty significant release coming up, so we'll keep things as they are for some weeks, we mitigated our worst cases by batching up the work in smaller pieces. I'll try out the upgrade somewhere along the following month. We'll have to watch out for the MSI security, we solved it using an undocumented feature in the previous ASB libs (#37). Might be that changed in the underlying connection string handling.

mookid8000 commented 2 years ago

Has anyone tried Rebus.AzureServiceBus 9.* with long-running message handlers (i.e. in need to automatically renewed peek locks)?