search

celery / kombu

Messaging library for Python.
http://kombu.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
2.86k stars 927 forks source link

py-amqp Too many heartbeats missed when using Gevent #1244

Closed jamieluckett closed 2 years ago

jamieluckett commented 4 years ago

We're currently using kombu in our product to read/write to RabbitMQ but we've been having issues with the heartbeats. The python program uses gevent to run 4 RabbitMQ connections at the same time with SimpleQueue objects for each connection. We're using .get(block=True) as well so that the greenlets wait for the next message to do work. In addition to these greenlets, we've also got an extra one running for each connection to maintain the heartbeats with connection.heartbeat_check.

With debug logging, I can see that this function is getting called every second for every connection, but one of the connections seems to not get back a heartbeat from RabbitMQ - which causes the connection to die. I've attached some logs, does anyone have any ideas?

(logs are for what is functionally identical to 2.5.2, I've just put in extra logging - same issue happens in latest code)

heartbeat_tick : for connection b7a4169be4c04c068f5ca87bc4faf610
heartbeat_tick : Prev sent/recv: 6/5, now - 6/5, monotonic - 2014.51239255, last_heartbeat_received - 1896.38198901 last_heartbeat_sent - 1957.38631076, heartbeat int. - 60 for connection b7a4169be4c04c068f5
ca87bc4faf610
heartbeat_tick : for connection b7a4169be4c04c068f5ca87bc4faf610
heartbeat_tick : Prev sent/recv: 6/5, now - 6/5, monotonic - 2015.463401421, last_heartbeat_received - 1896.38198901 last_heartbeat_sent - 1957.38631076, heartbeat int. - 60 for connection b7a4169be4c04c068f5ca87bc4faf610
heartbeat_tick : for connection b7a4169be4c04c068f5ca87bc4faf610
heartbeat_tick : Prev sent/recv: 6/5, now - 6/5, monotonic - 2016.416657776, last_heartbeat_received - 1896.38198901 last_heartbeat_sent - 1957.38631076, heartbeat int. - 60 for connection b7a4169be4c04c068f5ca87bc4faf610

2020-09-02 09:11:46,708 ERR [opsview.common.messagequeue.rabbitmq_kombu] Heartbeat error for '_state_changes_message_out_pump'
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/home/vagrant/projects/opsview-common/opsview/common/messagequeue/rabbitmq_kombu.py", line 145, in _heartbeat
    raise e
  File "/home/vagrant/projects/opsview-common/opsview/common/messagequeue/rabbitmq_kombu.py", line 141, in _heartbeat
    self.conn.heartbeat_check()
  File "/home/vagrant/projects/opsview-results-live/venv3/lib/python3.6/site-packages/kombu/connection.py", line 312, in heartbeat_check
    return self.transport.heartbeat_check(self.connection, rate=rate)
  File "/home/vagrant/projects/opsview-results-live/venv3/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
    return connection.heartbeat_tick(rate=rate)
  File "/home/vagrant/py-amqp/amqp/connection.py", line 762, in heartbeat_tick
    raise ConnectionForced('Too many heartbeats missed for connection {}'.format(self._connection_id))
amqp.exceptions.ConnectionForced: Too many heartbeats missed for connection b7a4169be4c04c068f5ca87bc4faf610
thedrow commented 4 years ago

Can you reproduce the same problem without gevent? What happens when you try eventlet? Can you reproduce the same error?

auvipy commented 2 years ago

closing for lack of feedback. will revisit upon report