Cannot apply recoverability policies when handler execution exceeds consumer ACK timeout leading to infinite message processing

[bording]

While #894 fixed the transport's connection recovery problems around the newly introduced consumer acknowledgement timeout, there is still a related problem.

Symptoms

When a handler runs longer than the timeout value, the message is requeued by the broker and the transport logs a warning but is unable to ACK the message due to the timeout. When this happens, the message will be retried until the handler completes quicker than the timeout. No configured recoverability policies will apply. In the extreme case when handler execution always takes longer than the timeout, the message undergoes infinite retries.

Who's affected

Affected are used who either:

• Have message handlers that always take long (by default more than 30 minutes) to execute. These messages undergo infinite retry attempts unless subsequent executions (after the first one) take less time. Customers who use the Outbox should are not affected because the second attempt bypasses the handler execution
• Have messages that fail due to a persistent issue (e.g. bug in code) and the handler execution time before the exception is raised multiplied by the number of immediate retries is higher than 30 minutes. Such messages undergo infinite retries causing constant resource pressure on the system

Root cause

RabbitMQ broker introduced the ACK timeout after which a message cannot be ACKed by the client.

Workaround

A temporary workaround for the issue, if you are affected and can't upgrade immediately, is to extend the consumer timeout value.

Backported to

• 6.1.3
• 7.0.2

[bording]

Fixed by #1071 and #1096

[bonieckimarcin]

I think we have found an issue #1259 related to fixes made in #1096 or #1071