ruby-amqp / bunny

Bunny is a popular, easy to use, mature Ruby client for RabbitMQ
Other
1.39k stars 303 forks source link

Channel callback from delivery acknowledgement timeout. #681

Closed dchompd closed 7 months ago

dchompd commented 7 months ago

Consumer silently losing its channel due to RabbitMQ closing the channel because of a delivery acknowledgement timeout.

I have a channel open with a consumer working it. Messages are being being pulled off and the consumer is manually acknowledging them. I ran into the issue of a delivery acknowledgement timeout (https://www.rabbitmq.com/docs/consumers#acknowledgement-timeout) that resulted in rabbit closing the consumers channel with a PRECONDITION_FAILED channel exception. This manifested in the application silently accumulating messages with no indication that the consumer was no longer being delivered messages on the channel. When I recreate the delivery ack timeout (create q with a short timeout and simulate a job that hangs the work pool thread and never acks), I see the AMQ::Protocol::Channel::Close frame (https://github.com/ruby-amqp/bunny/blob/8b700a00b62d4f887f53d4871f019d6d494c5f4d/lib/bunny/channel.rb#L1802) and also noticed there is the @on_error handler. Unfortunately, in this case, the on_error handler is never called (https://github.com/ruby-amqp/bunny/blob/8b700a00b62d4f887f53d4871f019d6d494c5f4d/lib/bunny/channel.rb#L1807-L1812) and the last_channel_error ends up pushed onto the continuations. If the thread working the message is hung, this error is never raised. I'm trying to understand if there is a way my channel can be notified that RabbitMQ is closing it due to the delivery acknowledgement timeout (reply code 406)? (besides monkey patching a condition into channel_level_exception_after_operation_that_has_no_response? https://github.com/ruby-amqp/bunny/blob/8b700a00b62d4f887f53d4871f019d6d494c5f4d/lib/bunny/channel.rb#L1822-L1824). My current monkey patch just adds an additional text to that line

method.reply_code == 406 && (method.reply_text =~ /unknown delivery tag/ || method.reply_text =~ /delivery acknowledgement on channel \d+ timed out/)

but I'm trying to fully understand what the best approach here would be. Thanks!