WebSocket route support

[ColtonMcInroy]

First of all, great job on Flow v10. It is a great tool and shows a lot of promise.

While playing with it, I went to create a new component that handles WebSocket routes but it appears that SOCKET routes do not work. I have been trying to look into how traffic is forwarded from the main process to a worker but I am unable to find it quickly so I figure it would be best to reach out.

Here is the example component I made to do some testing, it is a simple modification of the REST API -> Route but when creating the route, undefined is returned and attempting a WebSocket connection fails...

<script total>

    exports.name = 'Socket';
    exports.group = 'WebSocket';
    exports.icon = 'fa fa-road';
    exports.version = '1';
    exports.author = 'Colton McInroy';
    exports.config = { url: '/', timeout: 5000, limit: 5120, upload: false, send: 'all' };
    exports.outputs = [{ id: 'open', name: 'Open' }, { id: 'close', name: 'Close' }, { id: 'message', name: 'Message' }];

    exports.make = function(instance, config) {

        var prev = null;

        instance.configure = function() {

            if (prev) {
                prev && prev.remove();
                prev = null;
            }

            if (config.url) {

                var flags = [config.timeout];

                config.upload && flags.push('upload');
                console.log('Adding Route:', 'SOCKET ' + config.url);
                prev = ROUTE('SOCKET ' + config.url, function() {
                    var ctrl = this;
                    ctrl.autodestroy();
                    var msg = instance.newmessage();
                    var data;

                    switch (config.send) {
                        case 'payload':
                            data = ctrl.body;
                            break;
                        case 'query':
                            data = ctrl.query;
                            break;
                        case 'files':
                            data = ctrl.files;
                            break;
                        case 'params':
                            data = ctrl.params;
                            break;
                        case 'headers':
                            data = ctrl.headers;
                            break;
                        default:
                            data = { body: ctrl.body, query: ctrl.query, files: ctrl.files, user: ctrl.user, url: ctrl.url, headers: ctrl.headers, params: ctrl.params, ip: ctrl.ip };
                            break;
                    }
                    msg.refs.controller = ctrl;
                    ctrl.on('open', function(client) {
                        msg.refs.client = client;
                        msg.send('open', data);
                    });
                    ctrl.on('close', function(client) {
                        msg.refs.client = client;
                        msg.send('close', data);
                    });
                    ctrl.on('message', function(client, message) {
                        msg.refs.client = client;
                        data.message = message;
                        msg.send('message', data);
                    });
                    // msg.send('output', data);
                }, flags, config.limit);
                console.log('Route:', prev);
            }
        };

        instance.configure();
    };

</script>

<readme>
This component registers an HTTP Route and sends request data next. It stores `controller` instance in the `message.refs.controller` property. You must respond to the REST response component.

< __IMPORTANT__:<br>The component works only with a defined Proxy endpoint in the Flow settings.
< __URL__:<br>The URL must be relative to the defined Proxy endpoint. So if the endpoint is `/users/` and the desired address is `http://example.com/users/find` then the value must be `/find`

__Output data__:

``js
{
    "route": String,
    "params": Object,
    "query": Object,
    "body": Object,
    "files": Array,
    "headers": Object,
    "url": String,
    "ip": String
}
``
</readme>

<settings>
    <div class="padding">
        <div class="row" data-bind="flow.head.worker__hide:!value || flow.head.origin.length < flow.head.proxyurl.length">
            <div class="col-md-12 m">
                <div class="message message-error"><b><i class="fa fa-warning"></i>No proxy endpoint defined.</b><br>This component will not work until then. Please go to main screen, open settings of this FlowStream instance and set the Endpoint.</div>
            </div>
        </div>
        <div class="row">
            <div class="col-md-9 m">
                <div data---="input__?.url__required:1">Relative URL address</div>
                <div class="help">Relative path to: <span data-bind="flow.head.proxyurl__text:value?value:window.location.origin"></span></div>
            </div>
        </div>
        <div class="row">
            <div class="col-md-3 m">
                <div data---="input__?.timeout__required:1;type:number">Timeout</div>
            </div>
            <div class="col-md-3 m">
                <div data---="input__?.limit__required:1;type:number;ricon:!kB;align:1">Request limit</div>
            </div>
            <div class="col-md-3 m">
                <div data---="input__?.send__required:1;dirsource:all|Everything,payload|Payload,query|Query arguments,files|Files,params|Dynamic params,headers|Headers">Send to output</div>
            </div>
        </div>
    </div>
</settings>

<style>
    .CLASS footer { padding: 10px; }
    .CLASS footer span { font-family: var(--monospace); }
    .CLASS footer .method { background-color: var(--color); color: #FFF; padding: 2px 3px; border-radius: var(--radius); }
</style>

<body>
    <header>
        <i class="ICON"></i>NAME
    </header>
    <footer data-bind="CONFIG.method__show" class="hidden">
        <div><span class="method">SOCKET</span> <a data-bind="CONFIG.url__text__href:flow.head.proxyurl + (value[0] === '/' ? value.substring(1) : value)" target="_blank"></a></div>
    </footer>
</body>

[petersirka]

Hi @ColtonMcInroy, Thank you very much, we appreciate every feedback!

Currently WebSocket routing is not supported in worker/child_process mode. We don't have implemented WebSocket for the Total.js Proxy mechanism. Every request is routed/proxied by the Total.js framework to the child process. Hopefully, we will look into it soon. We plan to implement it.

But, you can disable child process mode (and it will work), just change /config file:

flowstream_worker    : false

With this option, all FlowStreams will be created in a single thread with the main process. If you do, then remove the proxy endpoint. Due to the fact that all routes will be combined with the main process, you must choose a unique WebSocket endpoint name.

[ColtonMcInroy]

Thanks for the quick response @petersirka. I have been programming with JS for about 20+ years at an enterprise level, although I am fairly new to the Total.JS framework, would you be able to point me to where to look for this proxy mechanism so that I could investigate and perhaps contribute a fix to implement this? I did notice there appears to be possible a unix socket involved with that being used to facilitate communication with workers but I would appreciate any information/details you could provide.

Or is this going to be something that is going to get addressed very soon and it's not worth me spending the time? I am building a library of components for Flow and would very much prefer to operate within the threaded environment but if this is going to be taken care of soon than I can build out the components with worker set to false for now. I would like to start setting up Flow to run in a production environment to replace lots of outdated code. Right now this socket issue is the only thing holding it back.

[petersirka]

I will look into it at least next week, OK? I hope that it will work during the weekend. We need it too, so this is a good time to implement it. I know what and where I need to change the code, so it will be much faster.

Additionally, we have updated our client-side library with Web components support (currently it's 100% backward compatible and with less code), and we plan to rewrite all components to a more modern and faster implementation in a short period of time. Maybe we can start tomorrow and next week you will have Flow prepared for several years. This is the last thing that needs to be implemented.

BTW: After 20 years, you must be a JS god. So I hope that you will be satisfied with your use of Total.js Flow.

[ColtonMcInroy]

Oh awesome, that is great to hear. I will wait then and use the suggested work around.

That is awesome, that is a great idea, and if you took a look at my other comment regarding embedding Flow into bigger applications I think making Flow itself a web component or something that is easily integrated into other applications would really make this tool flexible. I know I can see replacing many things with it.

Thanks... so far I can say that I am very impressed with Total.js Flow. I think there is always room for improvement, but this is a huge change compared to what has been out there. With the right components I think this could become a very big thing. Node-RED has had major limitations which they refuse to address, I know that there are many people trying to develop something similar to what you have done here, but I think so far this has beats all of them by a significant amount. Keep up the great work, and I hope that I will be able to help and contribute in the future in some way

[ColtonMcInroy]

I tried changing flowstream_worker : false But when I do this, I restart the Flow server, refresh the page, open a Flow and it gives me an error saying Not Authorized. I refresh the page again and I am faced with a login prompt which never allows me to login, when I log in with credentials it just refreshes the page. I do have a server extension installed on a couple flows which has the AUTH() implementation and I believe that when running without threads that AUTH implementation is executed for all calls including from the Flow application itself. This creates a conflict because now there are two AUTH implementations working against each other. The one which is part of Flow and the one within any Flows. I'll either have to temporarily eliminate mine for now to test sockets or just wait until you have completed the changes so I can test it.

[petersirka]

@ColtonMcInroy I have updated Total.js Proxy by adding support for WebSocket. You can test it in your Flow, just update Total.js framework $ npm install total4@beta.

[ColtonMcInroy]

@petersirka Thanks for the prompt work... I use yarn, so I stopped Flow... ran yarn add total4@beta then restarted Flow. The socket component was already in place as it came up, I can see that I get the same console result (from component above)... Adding Route: SOCKET / Route: undefined This shows that the ROUTE() call is returning undefined. I based this component on the REST API Route component and when it calls ROUTE it appears to be returned a reference which is then used here to remove the old route... if (prev) { prev && prev.remove(); prev = null; } This is the code responsible for cleaning up the route. I am able to connect to the websocket now though, but if I change the path, or the component is reconfigured the old socket currently cannot be removed.

[petersirka]

It was bug :-/. I have fixed it, please re-install beta version.

[ColtonMcInroy]

Perfect, it works now. Excellent work, thanks very much