search

rebus-org / Rebus.RabbitMq

:bus: RabbitMQ transport for Rebus
https://mookid.dk/category/rebus
Other
65 stars 45 forks source link

Message retried indefinitely due to unknown delivery tag error in RabbitMQ #122

Open simongullberg opened 1 month ago

simongullberg commented 1 month ago

Hi,

A few days ago, we encountered an issue that we haven't experienced before. A specific message was processed multiple times, and the exceptions we observed were as follows:

RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=406, text='PRECONDITION_FAILED - unknown delivery tag 5', classId=60, methodId=80
   at RabbitMQ.Client.Impl.SessionBase.Transmit(OutgoingCommand& cmd)
   at RabbitMQ.Client.Framing.Impl.Model.BasicAck(UInt64 deliveryTag, Boolean multiple)
   at Rebus.RabbitMq.RabbitMqTransport.<>c__DisplayClass55_0.<<Receive>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Rebus.Transport.TransactionContext.InvokeAsync(Func`2 actions)
   at Rebus.Transport.TransactionContext.Complete()
   at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ProcessMessage(TransactionContext context, TransportMessage transportMessage)

and

An error occurred when attempting to complete the transaction context

System.NullReferenceException: Object reference not set to an instance of an object.
   at Rebus.RabbitMq.RabbitMqTransport.<>c__DisplayClass55_0.<<Receive>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Rebus.Transport.TransactionContext.InvokeAsync(Func`2 actions)
   at Rebus.Transport.TransactionContext.Complete()
   at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ProcessMessage(TransactionContext context, TransportMessage transportMessage)

With each message redelivery, a new MessageId was generated, which leads me to believe the error tracker was unable to accurately track the error count. As a result, the message was retried indefinitely.

Is this a known issue, or do you have any insights into what might be causing this behavior?

Thank you!

mookid8000 commented 1 month ago

I have no idea what caused that particular error... but I would actually have expected that the message could be properly tracked, because the artificial message ID supplied when the message doesn't really have one is stable across message deliveries, because it's generated as a hash of the message body (see here for details).

Do you know where the message came from?