Support multiple serializers

[om26er]

Currently a single serializer needs to be provided to the WebSocketTransport class, it would be very useful to optionally allow to provide a list of serializers based on priority. If the router the "speaks" the first one it gets used, otherwise the next is tried.

It's more of an optimization but the binary serializers are much faster than json.

[konsultaner]

@om26er serializers are not negotiated in WAMP. Not sure if I will ever implement this. I will definitly accept a PR. If you need help with this, I could support you.

[om26er]

Right, we essentially need to send a list of serializers that our client talks in the initial handshake. The only user-facing change would be the addition of an API that takes serializers as a list instead of the current single serializer.

[konsultaner]

@om26er Could you get me more details on that? You want to extend the HELLO with a list of serializers and the CHALLENGE will tell the client what serializer to use?

[om26er]

@konsultaner the supported serializers have to be provided during the websocket handshake, they are negotiated before WAMP session is estabilished.

We need to change this code to take multiple serializers https://github.com/konsultaner/connectanum-dart/blob/master/lib/src/transport/websocket/websocket_transport_io.dart#L77

Once the connection is established the _socket.protocol can be queried to know which serializer was negotiated. Hence from that point onwards that serializer should be used for WAMP messaging.

The preference is from left to right, the router will return the first serializer that it supports.

[konsultaner]

@om26er _socket.protocol would have a value of wamp.2.json.msgpack.cbor, if all serializers are provided?

[konsultaner]

@oberstet does crossbar support that too? What do you think about it?

[om26er]

_socket.protocol would be wamp.2.json or wamp.2.msgpack or wamp.2.cbor depending on whichever sub-protocol (wamp.2.<serializer> in our case) is negotiated

This page might be of interest https://medium.com/@lancers/websocket-api-sec-websocket-protocol-subprotocol-header-support-277e34164537

Also afaik, Crossbar has that behavior and so does all autobahn libraries

[om26er]

And this line

_socket = await WebSocket.connect(_url, protocols: [_serializerType]);

would essentially look like

_socket = await WebSocket.connect(_url, protocols: [_serializerTypeCBOR, _serializerTypeMsgPack, _serializerTypeJson]);

[oberstet]

serializers are not negotiated in WAMP

yes, that is true. however, WAMP transports may have negotiation - as does WebSocket for example.

in code terms, both WAMP-WebSocket clients and servers maintain a list of serializers they support

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/websocket.py#L302

with WAMP-WebSocket clients, the client will send the list as WebSocket subprotocols: wamp.2.cbor, wamp.2.cbor.batched, ..

the WAMP-WebSocket server, that is the router, will select the first one from that list it supports

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/websocket.py#L198

RawSocket doesn't allow the client to provide a list for the server to choose from .. the client provides 1 serializer

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/twisted/rawsocket.py#L362

and the server must support that - otherwise it denies the client

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/twisted/rawsocket.py#L289

If the router the "speaks" the first one it gets used, otherwise the next is tried.

the application can do that if it feels to. none of the autobahn libraries have that built in .. I think;) in any case, it's largely uneeded in my eyes, because:

• your router supports all serializers, doesn't it? ;)
• the client can be optimized and hard-coded for the best one

[oberstet]

writing above .. reminds me of "batched transports" ... which is a thing, and can bring quite some efficiencies. in particular on TLS. because there you want to shuffle multiple WAMP messages in a single TLS segment ... and the way TLS Python Twisted interacts .. anyways, details;)

[konsultaner]

@om26er @oberstet thanks for all that input. I was blind about the fact, that the _socket.protocol already is an array... I'll fix that.

[konsultaner]

RawSocket doesn't allow the client to provide a list for the server to choose from .. the client provides 1 serializer

@oberstet at the moment there is only msgpack and json support for raw sockets. 0x0001 for json and 0x0010 for msgpack if we added 0x0100 for cbor, the sending 0x0111 could mean that all 3 serializers are supported.

@om26er do you need raw socket at all?

[oberstet]

@konsultaner There is no negotiation in RawSocket by design. KISS;) The client must announce the one serializer it wants to talk, and the router accepts or denies. That's it. No need for ANDing of multiple values.

Having said that, I guess we might lack definitions for RAWSOCKET_SERIALIZER_IDs in the spec for RawSocket? Are we indeed?

AutobahnPython and Crossbar.io support RawSocket over TCP, TLS, UDS, Pipes and Serial on all supported Serializers ("RAWSOCKET_SERIALIZER_ID"):

• json: 1
• msgpack: 2
• cbor: 3
• ubjson: 4
• flatbuffers: 5

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/twisted/rawsocket.py#L399

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/twisted/rawsocket.py#L528

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/wamp/serializer.py#L657

[konsultaner]

@oberstet @om26er ok, then well only have this feature for WebSockets. But to be honest, if crossbar supports all them why would some want to choose anyway?

the application can do that if it feels to. none of the autobahn libraries have that built in .. I think;) in any case, it's largely uneeded in my eyes, because:

• your router supports all serializers, doesn't it? ;)
• the client can be optimized and hard-coded for the best one

I don't really see a point. but I might do it if I get some time for that. @om26er If you are faster then me, feel free to send a PR.

[om26er]

I proposed a change to Nexus's WAMP client, which keeps the user-facing API as-is but internally offers a list of serializers to the router if user doesn't specifically request one https://github.com/gammazero/nexus/pull/270

I guess we can do a similar treatment here without changing the user API

[KSDaemon]

Let me put my 50cents:

• Indeed websocket protocol has a special header Sec-Websocket-Subprotocol that is used for serializer selection in WAMP.
• But there is no clear definition of how serializer is chosen. In our case client is initiator of the connection, so the client can put a list of supported serializers into Sec-Websocket-Subprotocol header (e.g. Sec-Websocket-Subprotocol: wamp.2.cbor, wamp.2.json. But what serializer will wamp router choose? First from the client list that is supported by router? Or first one from the router list that is supported by the client?
• There are more wamp routers besides Crossbar! :)

This is unclear, and every wamp router can act on it's own. For example Nexus Router choose serializer from it's own list. And JSON is on its first position. So if client sends wamp.2.cbor, wamp.2.msgpack, wamp.2.json. Nexus will always select wamp.2.json. In early days i even had to remove this feature from my wampy.js because Nexus always selects JSON instead of others.

So what i want to say: it is a good and in my mind useful feature. But we should clearly describe in WAMP specification that this is an ordered list, and that WAMP Router should use the first one supported from its side.

[oberstet]

• But what serializer will wamp router choose? First from the client list that is supported by router? Or first one from the router list that is supported by the client?

for wamp-websocket transports, which do use Sec-Websocket-Subprotocol, this follows from RFC6455 (websocket). in your example: when a client announces wamp.2.cbor, wamp.2.msgpack, wamp.2.json, the first one the router supports and wants to talk

But we should clearly describe in WAMP specification that this is an ordered list, and that WAMP Router should use the first one supported from its side.

for wamp-websocket, see RFC6455

for wamp-rawsocket: there is no support for selecting from multiple ones, see my comment https://github.com/konsultaner/connectanum-dart/issues/44#issuecomment-1142462072

[KSDaemon]

Well, I was also thinking so, but in RFC6455 (that i read a lot of course) it is stated like this: For client side:

|Sec-WebSocket-Protocol| header field, with a list of values indicating which protocols the client would like to speak, ordered by preference.

And for server side:

Either a single value representing the subprotocol the server is ready to use or null. The value chosen MUST be derived from the client's handshake, specifically by selecting one of the values from the |Sec-WebSocket-Protocol| field that the server is willing to use for this connection (if any).

the server is willing to use can be treated differently. So yes, server supports cbor, msgpack, json, client sent cbor, json. But server is willing to choose json even if it supports all client announced.

Well I would like to hope that every server choose the first one from client. But for example Nexus works differently :) But I will fix that to desired logic.