Allow for mappings between nodes (in bindings)

[rikkertkoppes]

You can currently create bindings between nodes (this is preferred), but there is no way to "map" one message to another. This may be desirable if you want to combine multiple streams to a single node and normalize the content. Or maybe only a part of the content is of interest to a particular node (and the rest a waste of bandwidth)

One approach that springs to mind is something along the way of bacon, which allows complicated stream merging and combining (e.g. creating a gate). This may be a bit over the top though.

Another approach is to allow a transducer to be defined on bindings. Bindings are always 1-to-1 as I understand, but multiple bindings merge into a stream to one node. Transducers could be directly coupled to a binding, since they are also 1-to-1. See http://jlongster.com/Transducers.js--A-JavaScript-Library-for-Transformation-of-Data for an in depth article.

Possible implementation (this would require server.conf.json to be a module, a bit like how grunt scripts are arranged):

var t = require('transducers');

module.exports = function()
    return {
        "port": 13900,
        "nodes": ["blib", "ping", "twitter", "controller", "proxy", "overlay"],
        "bindings": [
            {
                "from": "twitter",
                "to": "overlay",
                "pattern": "twitter:{add,remove}",
                "xform": t.map(function(data) {
                    //return a (renamed) subset of the data
                    return {
                        id: data.id,
                        message: data.text
                    };
                })
            },
            { "from": "controller", "to": "proxy" },
            { "from": "proxy", "to": "overlay" }
        ]
    };
};

On the other hand, you may also just refer to a transducer file somewhere (relative to the config.json), keeping everything plain json:

    {
        "port": 13900,
        "nodes": ["blib", "ping", "twitter", "controller", "proxy", "overlay"],
        "bindings": [
            {
                "from": "twitter",
                "to": "overlay",
                "pattern": "twitter:{add,remove}",
                "xform": "./transducers/maptweet.js"
            },
            { "from": "controller", "to": "proxy" },
            { "from": "proxy", "to": "overlay" }
        ]
    }

Where maptweet.js contains:

var t = require('transducers');

module.exports = t.map(function(data) {
        //return a (renamed) subset of the data
        return {
            id: data.id,
            message: data.text
        };
    })
};

In general, the following use cases spring to mind

• simple mapping (of course)
• filtering, only let through messages satisfying a particular predicate
• generating a running average or running sum of some value
• throttling a data flow, by letting only one through every second
• throttling by aggregating over a time interval
• aggregating batches of 3 messages

[poelstra]

Interesting ideas!

I'm wondering whether supporting 'user-defined' transducers etc directly in the server is a good idea from a stability stand-point. Maybe that type of thing is better suited to an external process, which listens to one node, transforms the data, then retransmits it to another node. That does add additional latency and bandwidth though, so maybe it should be present in the server.

One idea I was playing with, to make it easier for users to use, is to provide an mhub as a hosted service somewhere. In that case having pure configuration (not custom code) would be advantageous, and people will then probably have to use such an external process anyway.

Maybe some of the stuff you mention could still be provided by (configurable) 'standard' functionality, stuff that's built-in by default.

Especially the throttling/batching/filtering could probably be done quite easily.

Another transducer/filter I was thinking about, is a windowing function. E.g. to forward not just the last message, but the last X messages whenever a new one comes in, and/or a timeout expires.

I'll probably first add functionality to do this stuff to mclient, such that we can start experimenting with transforms, then move some of these to the server once they stabilize.

[rikkertkoppes]

Maybe a plugin system like browserify?

The hosted version could then simply not allow plugins.

[poelstra]

Not sure what you mean with 'plugin system like browserify'?

[poelstra]

Note that https://github.com/poelstra/mhub-relay now supports transforms.

[poelstra]

@rikkertkoppes Do you agree that mhub-relay and (in the very near future Node-RED) would be a better solution to this?

I suppose the only reason to have this directly in MHub would be to have lower latency (e.g. for transforms between match clock start events).

But as I mentioned before, it's also a significant risk for stability issues, which are harder to debug when they're inside MHub than in a standalone process, I suppose.

[rikkertkoppes]

Yeah, agree