search

celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

TypeError: no default __reduce__ due to non-trivial __cinit__ #356

Open emilhe opened 2 years ago

emilhe commented 2 years ago

I am trying to setup celery using the azureservicebus broker option. If I use redis (or rabbitmq) as the broker, my code works, but when I try to use azureservicebus I get the following error,

[2022-01-19 12:53:44,400: ERROR/MainProcess] Task handler raised error: TypeError('no default __reduce__ due to non-trivial __cinit__')
Traceback (most recent call last):
  File "/.../venv/lib/python3.9/site-packages/billiard/pool.py", line 596, in body
    put(task)
  File "/.../venv/lib/python3.9/site-packages/billiard/connection.py", line 233, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/.../venv/lib/python3.9/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
  File "stringsource", line 2, in uamqp.c_uamqp.CompositeValue.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__

I have debugged the issue and found that the error occurs when the billiard.reduction.ForkingPickler.dumps function is called on the azure.servicebus.ServiceBusReceivedMessage object. It this seems to be an issue with this object not supporting the serialization protocol. 

Ubuntu 20.04 (running under WSL2)
Python 3.9 (I also tried 3.7, same result)
celery=5.2.3
kombu=5.2.3
billiard=3.6.4.0

As a side note, I tested my connection string using the kombu examples, and they work. So I guess the issue is related to billiard (or maybe celery itself). I also posted the issue there.