search

celery / kombu

Messaging library for Python.
http://kombu.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
2.82k stars 920 forks source link

Kombu - amqp (cloudamqp) - Connection reset by peer #1138

Open emakarov opened 4 years ago

emakarov commented 4 years ago

Using kombu that is shipped with celery, in independent workers with asynchronous Hub.

Since August we didn't have any issues. But recently about 10 days ago started to experience continuous issues on production with following traceback when Producer was trying to publish: Connection reset by peer

In AMQP logs we saw this 

=WARNING EVENT==== Sat, 21 Dec 2019 19:07:14 GMT ===
2019-12-21 19:07:14.049 [warning] <0.14359.856> closing AMQP connection <0.14359.856> (52.221.88.206:22278 -> 10.54.9.92:5671, vhost: '***', user: '***'):
client unexpectedly closed TCP connection

Hard fixing the versions of libraries to (from latest):

amqp==2.5.1
celery==4.3.0
kombu==4.6.5

helped to fix the problem (at least looks like this) There's no client unexpectedly closed TCP connection logs in Rabbit anymore and workers sending messages as expected

At the same time, celery workers (which are independently used, had no issues)

auvipy commented 4 years ago

update with celery master?

emakarov commented 4 years ago

already switched to other library. kombu has many other issues, like wrong ordering of messages.

iglesiasroberto14 commented 1 year ago

This issue isn't solved. We are having the same problem with Celery when broker pool is enabled.

/ # pip freeze | grep -E '(kombu|amqp|celery)'
amqp==5.1.1
celery==5.2.7
kombu==5.2.4