search

crossbario / autobahn-python

WebSocket and WAMP in Python for Twisted and asyncio
https://crossbar.io/autobahn
MIT License
2.48k stars 766 forks source link

JSON auto-conversion for binary on Python 2 #574

Open oberstet opened 8 years ago

oberstet commented 8 years ago

This does work on Python 3 already, but NOT Python 2.

The WAMP spec describes a method to encode binary strings in JSON by prepending the base64 encoding of the binary string with \0. See here.

This isn't implemented. It would need to hook into the string processing with the JSON serializer.

More so, as we target PyPy anyway, we should probably aim for writing our own, bundled JSON thing:

We should have more unit tests regarding cross-serializer transparency. I think this is quite important, since it's one of the selling points of WAMP.

oberstet commented 8 years ago

Also see: https://github.com/crossbario/autobahn-js/issues/189

shredtek commented 8 years ago

I have also found that if I publish binary data with C++ and msgpack, subscribing with autobahn\python\json can stop published events from propagating. If I am subscribed with C++\msgpack I will receive the data, and then sometimes when I subscribe to the topic with autobahn\python\json I will stop receiving events in the c++\msgpack client. In either case I will not receive any events with the autobahn\python\json client. This does not seem deterministic and does not always happen, I am guessing some sort of race.

oberstet commented 7 years ago

Ok, implementing this on Python 2.7 requires monkey patching the stdlib, whereas on Python 3 it can be done cleanly: http://stackoverflow.com/questions/31869554/how-can-i-override-behavior-of-jsonencoder-for-binary-data

oberstet commented 7 years ago

@urbanpete AutobahnJS has integration tests (against Crossbar.io) for both CBOR and MessagePack:

These do work, but you will need to activate CBOR/MessagePack in AutobahnJS explicitly (due to this).

What will not work yet (because of this issue here) is AutobahnJS using JSON, and a different WAMP client using non-JSON with binary app payloads. The auto-conversion isn't implemented, but we'll add that.

oberstet commented 7 years ago

For other that run into this, here is the traceback that currently results when Crossbar.io tries to serialize a binary app payload for a WAMP client that is using JSON:

2017-04-04T22:50:42+0200 [Router      27984] WAMP message serialization error: Object of type 'bytes' is not JSON serializable
2017-04-04T22:50:42+0200 [Router      27984] Traceback (most recent call last):
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 119, in send
    payload, isBinary = self._serializer.serialize(msg)
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 93, in serialize
    return msg.serialize(self._serializer), self._serializer.BINARY
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/message.py", line 268, in serialize
    self._serialized[serializer] = serializer.serialize(self.marshal())
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 187, in serialize
    s = _dumps(obj)
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 162, in _dumps
    return json.dumps(obj, separators=(',', ':'), ensure_ascii=False)
  File "/home/oberstet/cpy361/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 95, in onMessage
    self._session.onMessage(msg)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/session.py", line 536, in onMessage
    self._router.process(self, msg)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/router.py", line 234, in process
    self._dealer.processYield(session, msg)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/dealer.py", line 691, in processYield
    self._router.send(invocation_request.caller, reply)
  File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/router.py", line 197, in send
    session._transport.send(msg)
  File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 123, in send
    raise SerializationError(u"WAMP message serialization error: {0}".format(e))
autobahn.wamp.exception.SerializationError: WAMP message serialization error: Object of type 'bytes' is not JSON serializable

One nice recent improvement though is that the client gets its in-flight call properly errored:

(index):57 targs: Array[1]0: "callee disconnected from in-flight request"length: 1__proto__: Array[0]error: "wamp.error.canceled"kwargs: Object__proto__: Object
(index):63 Connection lost: lost
oberstet commented 7 years ago

http://bugs.python.org/issue29992

oberstet commented 7 years ago

Ok, this is working now on Python 3 (both CPython 3 and PyPy3), merged via https://github.com/crossbario/autobahn-python/pull/808

It does NOT work on CPython 2, hence I leave this issue open as an enhancement.

See the comments in the unit test files and the serializer implementation, as well as above Python issue.

oberstet commented 5 years ago

as mentioned above, this works transparently for AB Py3 clients (and hence for CB), but not AB Py2:

    With the JSON serializer, this currently only works on Python 3 (both CPython3 and PyPy3),
    because even on Python 3, we need to patch the stdlib JSON, and on Python 2, the patching
    would be even hackier.

we should at least document that behavior - tbh, it's unlikely we find time for making AB Py2 work on that one.