mercadona / rele

Easy to use Google Pub/Sub
https://mercadonarele.readthedocs.io/en/latest/index.html
Apache License 2.0
200 stars 23 forks source link

Add subscription option to disable json parsing #229

Open csaroff opened 2 years ago

csaroff commented 2 years ago

Currently rele's subscription model only supports consuming messages that are json deserializable.

Unfortunately some of GCP's core services publish non-json messages. For example, the gcp dicom store publishes notifications to a topic when a new dicom instance was inserted, but the message field is just the dicom instance path. When attempting to consume these messages through rele, we get the following error.

Exception raised while processing message for dicom_uploads - handle_upload: JSONDecodeError
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/rele/subscription.py", line 112, in __call__
    data = json.loads(message.data.decode("utf-8"))
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Can we add an option to disable deserialization of these messages? Something like:

from rele import sub

@sub(topic='my-topic', deserialize_message=False):
def handle_upload(data, **kwargs):
    print('Handling data', data)
andrewgy8 commented 2 years ago

Another extension to this is something where the Callback deserialize function could be configurable.

Something like:

def deserialize_func(message):
    return message

@sub(topic='my-topic', deserialize=deserialize_func):
def handle_upload(data, **kwargs):
    print('Handling data', data)

By default, I would expect the current behaviour. But for some use cases, like @csaroff 's , you may want to control the deserialization logic yourself.

csaroff commented 2 years ago

Yeah, agreed. That's much better