Reopened: "Received 0x01 while expecting 0xce" errors when using python-amqp

[mskogorevrmc]

I previously opened a Issue - https://github.com/celery/celery/issues/6232, I was advised to update celery, but the problem remained. Could someone please help me sort this out?

On Debian Linux with Python3.5 with these packages:

celery                 4.4.6
billiard               3.6.3.0
kombu                  4.6.11
amqp                   2.6.1

I use RabbitMQ as brocker. I occasionally get the following errors:

 Received 0x00 while expecting 0xce
Traceback (most recent call last):
  File "/var/www/my_app/app/mod_axmsg/dispatch.py", line 376, in execute
    result_value = func_meta.func(*task.args_list, **task.kwargs_dict)
  File "/var/www/my_app/app/mod_rmc/libs/health.py", line 91, in update_all_health
    health.update_all_health()
  File "/var/www/my_app/app/mod_core/libs/health.py", line 515, in update_all_health
    result.join()
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 772, in join
    disable_sync_subtasks=disable_sync_subtasks,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 228, in get
    on_message=on_message,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 193, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 260, in _wait_for_pending
    on_interval=on_interval):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/rpc.py", line 63, in drain_events
    return self._connection.drain_events(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/connection.py", line 323, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 505, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 510, in blocking_read
    frame = self.transport.read_frame()
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/transport.py", line 280, in read_frame
    'Received {0:#04x} while expecting 0xce'.format(ch))
amqp.exceptions.UnexpectedFrame: Received 0x00 while expecting 0xce

----------------------------------
----------------------------------

 Received 0x50 while expecting 0xce
Traceback (most recent call last):
  File "/var/www/my_app/app/mod_axmsg/dispatch.py", line 376, in execute
    result_value = func_meta.func(*task.args_list, **task.kwargs_dict)
  File "/var/www/my_app/app/mod_rmc/libs/health.py", line 91, in update_all_health
    health.update_all_health()
  File "/var/www/my_app/app/mod_core/libs/health.py", line 570, in update_all_health
    result.join()
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 772, in join
    disable_sync_subtasks=disable_sync_subtasks,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 228, in get
    on_message=on_message,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 193, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 260, in _wait_for_pending
    on_interval=on_interval):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/rpc.py", line 63, in drain_events
    return self._connection.drain_events(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/connection.py", line 323, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 505, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 510, in blocking_read
    frame = self.transport.read_frame()
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/transport.py", line 280, in read_frame
    'Received {0:#04x} while expecting 0xce'.format(ch))
amqp.exceptions.UnexpectedFrame: Received 0x50 while expecting 0xce

----------------------------------
----------------------------------

 Received 0x39 while expecting 0xce
Traceback (most recent call last):
  File "/var/www/my_app/app/mod_axmsg/dispatch.py", line 376, in execute
    result_value = func_meta.func(*task.args_list, **task.kwargs_dict)
  File "/var/www/my_app/app/mod_rmc/libs/health.py", line 91, in update_all_health
    health.update_all_health()
  File "/var/www/my_app/app/mod_core/libs/health.py", line 570, in update_all_health
    result.join()
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 772, in join
    disable_sync_subtasks=disable_sync_subtasks,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/result.py", line 228, in get
    on_message=on_message,
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 193, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 260, in _wait_for_pending
    on_interval=on_interval):
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/celery/backends/rpc.py", line 63, in drain_events
    return self._connection.drain_events(timeout=timeout)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/connection.py", line 323, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 505, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/connection.py", line 510, in blocking_read
    frame = self.transport.read_frame()
  File "/usr/local/my_app/lib/python3.5/site-packages/amqp/transport.py", line 280, in read_frame
    'Received {0:#04x} while expecting 0xce'.format(ch))
amqp.exceptions.UnexpectedFrame: Received 0x39 while expecting 0xce

There are other tickets, but there is no activity in them: https://github.com/celery/celery/issues/1779 and https://github.com/celery/celery/issues/2066

[ShaheedHaque]

Superficially at least, this has some similarity to #5605, but a deep dive into the code/debugging would be needed to be sure.

[thedrow]

I'm not sure how to help without a test case to reproduce the problem consistently.

[auvipy]

need to know current status of it