search

Azure / azure-service-bus-dotnet

☁️ .NET Standard client library for Azure Service Bus
https://azure.microsoft.com/services/service-bus
Other
235 stars 120 forks source link

Lock renewal fails if the time on the receiving computer is set incorrectly #688

Open marcinkaluza opened 5 years ago

marcinkaluza commented 5 years ago

Actual Behavior

We are using default lock duration of 1 minute with auto lock renewal and auto complete. The MaxAutoRenewDuration is set to 3 hrs. If the receiving computer has incorrect time set (late by say 10+ seconds), automatic lock renewal fails with MessageLockLostException as it is fired too late. This is because the time for automatic renewal is determined by MessagingUtilities.CalculateRenewAfterDuration which substracts the DateTime.UtcNow from message.SystemProperties.LockedUntilUtc. If the local time is off and the calculated difference is too big, it leads to lost lock and the message is being redelivered elsewhere. One could argue that we should have the time set correctly, but it is beyond our control as the app runs in an Azure hosted Kubernetes cluster.

Expected Behavior

The time to execute lock renewal should be calculated independently of local clock. This could be done using LockDuration property of the underlying queue as opposed to the message.SystemProperties.LockedUntilUtc

Versions

axisc commented 5 years ago

@marcinkaluza - 10 seconds is probably too high for a time skew - a few milliseconds (or a couple seconds may be ok).

Have you raised a support ticket with AKS?

SeanFeldman commented 5 years ago

I've seen this with my customers. They had lock duration set to 30 seconds and time skew of 37 seconds. The servers have to be time synched. In their case, the azure firewall around their infrastructure was preventing from the sync to take place and the time skew has grew over time.

SeanFeldman commented 5 years ago

Reminds me of another issue that would benefit from dealing with only one clock by using TimeSpan rather than DateTime.