When Native Delayed delivery is enabled in version 1.2 of the transport, the timeout manager is still enabled. Both features rely on a sub-queue with the same naming convention endpointName.Timeouts.
Unfortunately, this means that messages destined for the native delayed delivery feature may be processed by the timeout manager and vice-versa. Both of these will throw an exception in this case, triggering the retry mechanism. This may allow the correct feature to pick up the message, or the message may be reprocessed by the wrong feature again.
Eventually, these messages may be sent to the configured error queue.
Expected behavior
Messages configured with delayed delivery should be dispatched to the correct queue at the desired time.
Actual behavior
Messages configured with delayed delivery may be retried multiple times and dispatched to error queue. Retrying the message from the error queue may not work and the message may be returned to the error queue.
Versions
Please list the version of the relevant packages or applications in which the bug exists.
NServiceBus.Transport.Msmq 1.2.x
NOTE: Version 2.x of the transport targets NServiceBus version 8.x, which does not contain the timeout manager.
Steps to reproduce
Create a new NServiceBus endpoint with version NServiceBus.Transport.Msmq 1.2.x
Send a large batch of messages with a short delayed delivery timeout. These will be sent via the timeout manager as native delayed delivery is not enabled.
Stop the endpoint before the timeout manager has had time to process all of the messages in the .timeouts queue
Enabled native delayed delivery and restart the endpoint.
Send a large batch of messages with a short delayed delivery timeout
Wait for the delayed delivery period and check the logs and error queue.
Look for many instances of:
This message does not represent a timeout from the native delayed delivery
Non timeout message arrived at the timeout manager, id: from the timeout manager
Check the error queue for messages which have failed to be delivered.
Relevant log output
From timeout manager:
2024-06-23 17:48:26,8161 [INFO] [NServiceBus.RecoverabilityExecutor] (null): Immediate Retry is going to retry message 'd724bce6-afe5-4894-a267-b19800f404bf' because of an exception:
System.InvalidOperationException: Non timeout message arrived at the timeout manager, id:d4275853-2ef5-460d-a6cc-88df0353d417\344710957
at NServiceBus.StoreTimeoutBehavior.<Invoke>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.TransportReceiver.<InvokePipeline>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NServiceBus.TransportReceiver.<InvokePipeline>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.Msmq.ReceiveStrategy.<TryProcessMessage>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.Msmq.TransactionScopeStrategy.<ProcessMessage>d__2.MoveNext()
Additional Information
Workarounds
Choose one of the following workarounds:
Upgrade to version 2.x of the transport, enable native delayed delivery, and migrate existing timeouts
Do not enable native delivery on version 1.2 of the transport. Rely on timeout manager.
If the native delayed delivery feature has been enabled, then messages may be found in the .timeouts queue which the timeout manager cannot process. These messages will be sent to the error queue with the stack trace noted above in the log details.
Contact Particular Support for assistance in retrying these messages successfully.
Describe the bug
Description
When Native Delayed delivery is enabled in version 1.2 of the transport, the timeout manager is still enabled. Both features rely on a sub-queue with the same naming convention
endpointName.Timeouts.Unfortunately, this means that messages destined for the native delayed delivery feature may be processed by the timeout manager and vice-versa. Both of these will throw an exception in this case, triggering the retry mechanism. This may allow the correct feature to pick up the message, or the message may be reprocessed by the wrong feature again.
Eventually, these messages may be sent to the configured error queue.
Expected behavior
Messages configured with delayed delivery should be dispatched to the correct queue at the desired time.
Actual behavior
Messages configured with delayed delivery may be retried multiple times and dispatched to error queue. Retrying the message from the error queue may not work and the message may be returned to the error queue.
Versions
Please list the version of the relevant packages or applications in which the bug exists.
NOTE: Version 2.x of the transport targets NServiceBus version 8.x, which does not contain the timeout manager.
Steps to reproduce
.timeoutsqueueLook for many instances of:
This message does not represent a timeoutfrom the native delayed deliveryNon timeout message arrived at the timeout manager, id:from the timeout managerCheck the error queue for messages which have failed to be delivered.
Relevant log output
From timeout manager:
Additional Information
Workarounds
Choose one of the following workarounds:
Possible solutions
Additional information
If the native delayed delivery feature has been enabled, then messages may be found in the
.timeoutsqueue which the timeout manager cannot process. These messages will be sent to the error queue with the stack trace noted above in the log details.Contact Particular Support for assistance in retrying these messages successfully.