search

gmr / rabbitpy

A pure python, thread-safe, minimalistic and pythonic RabbitMQ client library
http://rabbitpy.readthedocs.org
BSD 3-Clause "New" or "Revised" License
242 stars 58 forks source link

NoAck Consumers get socket closed by rabbitmq due to lack of heartbeat #98

Closed tom-railpod closed 7 years ago

tom-railpod commented 8 years ago

Versions: rabbitpy 0.27.1 RabbitMQ 3.2.4. on ubuntu 14.04.4

The failure was caused by a connection with only consumers (without an ACK). This resulted in no back traffic from the consumer to rabbitmq. After the the timeout period, rabbitmq closed the socket and the consumer died.

Note: As a prior workaround, the connection was created with a timeout of 0 which should have turned off the timeout. rabbitmq appeared not to honor this as the timeout was listed as 520 seconds.

The fix was to a single rabbitpy file: channel0.py. It was altered to construct and send a heartbeat message back to rabbitmq on the timeout interval.

The production GitHub repo of rabbitpy 0.27.1 (https://github.com/gmr/rabbitpy) was forked to: (https://github.com/tom-railpod/rabbitpy) and the changes were made to a single file (channel0.py)

The only enhancement I might suggest is to only send the heartbeat if there is no data has been recently sent back to rabbitmq. I was under production delivery pressure so could not adequately pursue this. On brief examination, I didn't find a "num_of_bytes_sent" counter and didn't really have a way to test it. There is probably no performance issue with the code as is.

I've generated a pull request

ghost commented 8 years ago

Funny story, I spent hours finding out this exact same problem in another Python library and came to that conclusion today also. I can confirm that RabbitMQ 3.6.0 & 3.6.1 exhibit this behavior and this sort of patch seems necessary, though I have not use this library.

eandersson commented 8 years ago

I think the solution based on an irc conversation would be to send heartbeats from the client if no data has been sent or received within n/2 of the interval.

ghost commented 8 years ago

I concur. I just interacted with your library in a way that had unintended consequences. Maybe I was using the wrong connection for the heartbeat?

On 6/23/2016 11:17 AM, Erik Olof Gunnar Andersson wrote:

I think the solution based on an irc conversation would be to send heartbeats from the client if no data has been sent or received within n/2 of the interval.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gmr/rabbitpy/issues/98#issuecomment-228083956, or mute the thread https://github.com/notifications/unsubscribe/ATKoWZaPaNT3BBA6nOMif-ZWyBup7Kfqks5qOqOhgaJpZM4I7_sQ.

tom-railpod commented 8 years ago

"send heartbeats from the client if no data has been sent or received within n/2 of the interval."

That wouldn't work on the case that failed. There was tons of data coming FROM rabbitmq to the rabbitpy consumer but it was unack'ed. There was just no data going BACK from rabbitpy queue to rabbitmq.

I actually looked (unsuccessfully) for a rabbitmq spec on the heartbeat. If it is as you suggested, then rabbitmq is buggy. I think the right answer is what I mentioned in initial post, i.e. send heartbeat back to rabbitmq if no data has been sent BACK in n/2 of the interval.

"I just interacted with your library in a way that had unintended consequences" What were the consequences? The only thing I was really worried about was an infinite round-robin echoing of heartbeats as there is code in rabbitpy which echos a heartbeat if if gets one.

ghost commented 8 years ago

Sorry about talking about interacting with a library, thought this was a different source. I was talking about amqp-storm with the unintended consequences.

If you note, the RabbitMQ documentation says that the 'official'-ish .Net and Java clients send heartbeats every interval/2 seconds, which is another strong argument in favor of that behavior. https://www.rabbitmq.com/heartbeats.html

On 6/23/2016 1:35 PM, tom campbell wrote:

"send heartbeats from the client if no data has been sent or received within n/2 of the interval."

That wouldn't work on the case that failed. There was tons of data coming FROM rabbitmq to the rabbitpy consumer but it was unack'ed. There was just no data going BACK from rabbitpy queue to rabbitmq.

I actually looked (unsuccessfully) for a rabbitmq spec on the heartbeat. If it is as you suggested, then rabbitmq is buggy. I think the right answer is what I mentioned in initial post, i.e. send heartbeat back to rabbitmq if no data has been sent BACK in n/2 of the interval.

"I just interacted with your library in a way that had unintended consequences" What were the consequences? The only thing I was really worried about was an infinite round-robin echoing of heartbeats as there is code in rabbitpy which echos a heartbeat if if gets one.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gmr/rabbitpy/issues/98#issuecomment-228124120, or mute the thread https://github.com/notifications/unsubscribe/ATKoWQQIkTlPSAxHX4s3aNtUxj8xCBAVks5qOsPfgaJpZM4I7_sQ.

gmr commented 7 years ago

This is addressed in 1.0