MQTT: payload mapping

[oberstet]

I think this should be configurable on realms and per URI (patterns), similar to this

mqtt_payload_format == opaque | json | cbor | ..

With `mqtt_payload_format==opaque', payload from MQTT publisher is published using payload transparency to WAMP side (we need that exposed .. dig through the code here).

When a WAMP publisher publishes to a topic configured for opaque payload on MQTT, it'll be just forwarded "as is" to the MQTT client. If a WAMP publisher does a regular publish to such a topic, the router will log a warning and throw away the event (since it doesn't know how to serialize).

With mqtt_payload_format==json, the MQTT client MUST publish payload serialize in JSON with content

{
   "args": [],
   "kwargs": {}
}

If it does publish with nonconforming payload, deny the publish (QoS1/2) or silently throw away the event.

The other direction is straigt forward: WAMP publisher publishes event, event gets serialized to JSON like above for MQTT.

[hawkowl]

I think this is the next big important thing... will have to work on exposing it tomorrow. Unsure how this will work, as it'll need, like, two functions passed to subscribe -- one which takes args/kwargs (and is also used by cryptobox), and one which takes non-cryptobox transparent payloads. What do you think @oberstet?

[hawkowl]

it's either that or two subscriptions, one which takes args/kwargs and one which accepts transparent payloads... but that seems kinda silly to me.

[oberstet]

Leaving cryptobox out for a moment, here is a sketch / start for discussion:

1. MQTT client publishes event to topic T:

• if the router has T configured to be mqtt_payload_format == opaque, then a WAMP event with args=[<opaque_payload>] and kwargs=None is dispatched by the broker (where <opaque_payload> is a binary string - means we finally have to implement transparent base64 encoding for WAMP/JSON too;) The WAMP spec talks about this)
• if the router has T configured to be mqtt_payload_format == json|cbor|.., then the broker deserializes the raw payload according to the configured serializer, and will expect a dict with args and kwargs, and use these to dispatch a WAMP event
• if the latter process of transformation fails for some reason, the event publication is denied (either silently (publish QoS 0) or with ERROR (publish QoS 1/2)

2. MQTT client subscribed to T receives an event:

• if the router has T configured to be mqtt_payload_format == opaque, then args[0] is expected to be of type "binary string", and that is dispatched to the MQTT client as payload
• if the latter process of transformation fails, eg args==None or type(args[0]) != binary, then we might log a warning, dispatch to WAMP clients, but skip dispatching to MQTT clients
• if the published event also has kwargs and/or args[i] for i > 0, then these bits are lost for MQTT subscribers
• if the router has T configured to be mqtt_payload_format == json|cbor|..., then a dict {"args": .., "kwargs": .., "details": ...} is serialized according to the serializer configured, and the resulting binary string is the MQTT event payload

[oberstet]

@goeddea ^

[oberstet]

The dict used for transformation could look like:

{
   "args": ["positional args of event"],
   "kwargs": {"keyword args of event"},
   "details": {"details of event, such as publisher_authid etc if applicable"}
}

@hawkowl I think this is similar of what we do in the REST bridge? I think we should make it similar / same if possible.

[hawkowl]

Should the payload format go on the topic? I was thinking it would go on the adapter... I'm not sure.

[oberstet]

Do we want to support this use case?

A WAMP aware application using a plain MQTT library to publish full featured WAMP events via MQTT. Effectively, "tunneling". Eg, MQTT only has a single binary string as payload. WAMP has richer semantics, up to positional and keyword based parameters and return values.

[markos]

In my use case, I have an existing infrastructure using MQTT as protocol, and I want to be able to connect a newly developed system that uses WAMP + Crossbar. I cannot/do not want to change the MQTT communication, but I do like the idea of having both transports being able to receive the payloads. MQTT protocol in my case uses manual JSON serialization/deserialization, so the payload sent over WAMP could be constructed manually on the publisher. My guess is that I should use the opaque payload_method but this was failing as kwargs was null, and the PR #983 tried to fix this but it was wrong. I do think that the underlying logic was right though, each transport in Crossbar should perform the necessary translations automatically, so that the receiver would receive the exact same message as if it were sent from an MQTT publisher (and ditto for a WAMP receiver).

[oberstet]

Actually, we now have three configurable methods of payload mapping which should allow to address most use cases, pls see https://github.com/crossbario/crossbar-examples/tree/master/mqtt/basic

This is on master branch, will be in next release .. closing