ruby-amqp / bunny

Bunny is a popular, easy to use, mature Ruby client for RabbitMQ
Other
1.39k stars 303 forks source link

Dangling Bunny::Sessions Refusing to Be Closed #586

Closed nwochaadim closed 5 years ago

nwochaadim commented 5 years ago

We are publishing RabbitMQ events inside a background job backed by DelayedJob and AWS SQS. Our publish code that runs inside the job looks like this:

    exchange.publish(payload, opts)
    connection.close

where exchange is a Bunny::Exchange and connection is a Bunny::Session

Here, we explicitly close the connection as soon as we are done with publishing the message. However, using the RabbitMQ dashboard, we can still see these idle connections with no channels. Meaning the channels somehow got closed, however, the session persisted

Prior to moving this code inside a background job, we weren't seeing this issue. So it has to do with something with the Background jobs and Bunny

Any insight into this problem will be really useful

michaelklishin commented 5 years ago

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details. At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Please post this to rabbitmq-users.

Thank you.

michaelklishin commented 5 years ago

I cannot suggest much with the amount of information provided.

All connection events are logged by RabbitMQ, including connection closure, abrupt closure (TCP connection loss or termination) and alarms.

Short lived connections is not how RabbitMQ protocols are supposed to be used and at a certain rate such usage will lead to a well known and well understood problem of high connection churn.

Your connections may or may not enable heartbeats (they are on by default) or use a high value. Contrary to the popular belief, TCP connection loss is not detected immediately by the peer. The Heartbeats guide covers this.

Lastly, it can be an issue in the management plugin. rabbitmq-diagnostics list_connections is the most "up to date" way of listing connections. I cannot suggest anything else since you haven't provided any RabbitMQ version information.

michaelklishin commented 5 years ago

Also, by far the most extensive source of information about what's going on with a connection is a traffic capture, which can also be performed on either side using tcpdump on the command line.

michaelklishin commented 5 years ago

While unrelated, the code snippet posted here is asking for data loss as it closes the connection after publishing without waiting for a confirmation from RabbitMQ. See