Support mapping arbitrary message payloads in AMQP-bridge

[thjaeckle]

The AMQP-bridge currently expects that all messages it receives are already in Ditto Protocol (JSON). That means that if Ditto connects for example to Eclipse Hono, the AMQP-bridge can only make sense of data sent in our defined protocol.

That is a problem for various reasons:

• the Ditto Protocol is not designed to be lightweight - as JSON and with the included data it is quite verbose
• devices connecting to Eclipse Hono and sending their data e.g. via MQTT should not be aware of the Ditto Protocol - they should just send "what" data should be changed to "which" value

Ditto therefore wants to support a first easy "mapping" of those payloads:

• new mappings should be added dynamically at runtime (without the need to restart the AMQP-bridge)
• the mapping should follow the mantra: "keep simple things simple, make hard things possible"
• mappings must be executed "sandboxed": as the mappings could be defined in a programming language, it must be ensured that they don't interfere with the operation of the AMQP-bridge service or any other mappings

Our idea is to define mappings in JavaScript which is executed inside the JVM with as much sandboxing as possible.

Any other ideas?

[danielFesenmeyer]

What do you think about applying these mappings to the websocket channel also? It might be a bit easier and faster to test the mappings.

[danielFesenmeyer]

I would suggest to provide the JSONata library in our Javascript sandbox. It seems to be a very powerful helper for JSON transformations. The user should decide whether she uses JSONata or "plain Javascript" (or a combination?).

JSONata is used/provided by several widely used tools such as Node-RED and ELK stack. And there is an online JSONata exerciser for trying out mappings: http://try.jsonata.org/

[thjaeckle]

Adding mappings to WS could be beneficial as well, but I wouldn't do it in the first step. Testing the mappings is IMO a task for a separate tool / web application (e.g. a simple html+javascript you can run locally).

I already had a look at JSONata because I also think it's an awesome tool to work with JSON data. But I had some issues getting it to work in a Java environment (e.g. via Rhino engine) as the library was just too big and also had dependencies.

And basically this would just be "syntactic sugar" for working with special payloads (JSON based) - there is nothing you can't do with plain JavaScript ;) JavaScript also works quite well with JSON ;)

[geglock]

I think that transformations for the WS channel could be beneficial, but I consider them less important. The WebSocket channel is mainly targeted at Web development or "capable" devices/gateways and these can do transformations/mappings on their side quite easily. The AMQP channel is the main channel for massive device connections (e.g. via Eclipse Hono) and especially for devices with small footprints which cannot do transformations on their own.

Regarding JSONata: I also noted that it is quite "big" (~50k javascript minified). But at least it does not contain any dependencies (see https://github.com/jsonata-js/jsonata/blob/master/package.json). For complex json-to-json transformations it would a good option. In general I consider "template"-based approaches as nice but not essential. They are normally meant as option to define logic when no usage of scripting is possible. But in Ditto we plan to provide scripting as an option and so there is no hard requirement.

[ameinhardt]

I'm also interested in this transformation, in order to being able to use PPMP (see specification) to ditto format. I'd be happy to try out the java or javascript transformation.

Regarding JSONata: in my experience, the devDependency babel-plugin-transform-runtime is not pure dev, but actually adds it's code to the output compilation. That version still includes babel-runtime includes regenerator-runtime includes facebook regenerator still with PATENTS clause, which might not be appropriate for every company. Only babel 7 resolves to MIT-licensed transpiler

[thjaeckle]

Resolved by PR #141