search

earlgreyness / aio-celery

Celery worker for running asyncio coroutine tasks
22 stars 3 forks source link

pathlib types cannot be used as task args -> "Object of type PosixPath is not JSON serializable" #2

Open rkrell opened 7 months ago

rkrell commented 7 months ago

Hello,

first, I want to thank you for this project, after struggling with calling async tasks in celery using an alternative pool.

I have an issue here when using a base_path: pathlib.Path object as argument to a task:

Traceback (most recent call last):
  File "/home/rkrell/work/project/backend/app/api/routers/metadata.py", line 37, in metadata
    res: AsyncResult = await metadata_task.delay(barcode, base_path)
  File "/home/rkrell/.pyenv/versions/project-pypy3.10-7.3.13/lib/pypy3.10/site-packages/aio_celery/annotated_task.py", line 55, in delay
    return await self.apply_async(args=args, kwargs=kwargs)
  File "/home/rkrell/.pyenv/versions/project-pypy3.10-7.3.13/lib/pypy3.10/site-packages/aio_celery/annotated_task.py", line 40, in apply_async
    return await self.app.send_task(
  File "/home/rkrell/.pyenv/versions/project-pypy3.10-7.3.13/lib/pypy3.10/site-packages/aio_celery/app.py", line 166, in send_task
    create_task_message(
  File "/home/rkrell/.pyenv/versions/project-pypy3.10-7.3.13/lib/pypy3.10/site-packages/aio_celery/amqp.py", line 53, in create_task_message
    body: bytes = json.dumps(
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/__init__.py", line 237, in dumps
    return _default_encoder.encode(obj)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 190, in encode
    self.__encode(o, markers, builder, 0)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 230, in __encode
    self.__encode_list(o, markers, builder, _current_indent_level)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 254, in __encode_list
    self.__encode(elem, markers, builder, _current_indent_level)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 230, in __encode
    self.__encode_list(o, markers, builder, _current_indent_level)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 254, in __encode_list
    self.__encode(elem, markers, builder, _current_indent_level)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 238, in __encode
    res = self.default(o)
  File "/home/rkrell/.pyenv/versions/pypy3.10-7.3.13/lib/pypy3.10/json/encoder.py", line 174, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable

In celery, I used the pickle protocol for serializing, which avoided this.

Maybe it would not be bad to offer also pickle as an alternative serialization approach, which would be a clean way to solve this without making workarounds for not JSON-serializable types.