search

celery / kombu

Messaging library for Python.
http://kombu.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
2.85k stars 926 forks source link

YAML not support memoryview #890

Open wencan opened 6 years ago

wencan commented 6 years ago

Hello I try to implement a celery protocol client based on kombu. When the result content_type is yaml, kombu throws an error:

  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
    return decode(data)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 24, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
AttributeError: 'memoryview' object has no attribute 'read'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/kombu/messaging.py", line 618, in _receive_callback
    decoded = None if on_m else message.decode()
  File "/usr/local/lib/python3.5/dist-packages/kombu/message.py", line 192, in decode
    self._decoded_cache = self._decode()
  File "/usr/local/lib/python3.5/dist-packages/kombu/message.py", line 197, in _decode
    self.content_encoding, accept=self.accept)
  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
    return decode(data)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.5/dist-packages/vine/five.py", line 178, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.5/dist-packages/kombu/serialization.py", line 263, in loads
    return decode(data)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 24, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
kombu.exceptions.DecodeError: 'memoryview' object has no attribute 'read'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kombu_.py", line 91, in <module>
    recv(task_id)
  File "kombu_.py", line 85, in recv
    conn.drain_events(timeout=0.1)
  File "/usr/local/lib/python3.5/dist-packages/librabbitmq/__init__.py", line 227, in drain_events
    self._basic_recv(timeout)
SystemError: <built-in method _basic_recv of Connection object at 0x7f29e74f4a68> returned a result with an error set

Finally, I replaced the yaml decoder:

import kombu.serialization

def yaml_load(data, *args, **kwargs):
    if isinstance(data, memoryview):
        data = data.tobytes()
    return yaml.safe_load(data, *args, **kwargs)

kombu.serialization.registry.register('yaml', yaml.safe_dump, yaml_load,
                                      content_type='application/x-yaml',
                                      content_encoding='utf-8')
ciprianiacobescu commented 3 years ago

Had the same problem, the solution worked for me.

Celery 3.x Kombu 3.x PyYaml 3.x

thedrow commented 3 years ago

We'd happily accept a PR to fix this issue.