WebRTC or WebSockets interface

[psychowood]

I'm trying to undestand if I can develop a web-based, client only, solution to implement real-time control (e.g. audioreactive/camera) to WLED.

Problem is, there's no way to send UDP packets from javascript.

Describe the solution you'd like Would you consider adding a WebRTC interface reusing the UDP protocol?

Describe alternatives you've considered AFAIK the HTTP and JSON interfaces are not enough to allow realtime control of single leds. This could be integrated in the mobile apps without changes to WLED, but I'd keep that as a second option.

Additional context Ideally, a client only solution could be easily integrated in the WLED web-ui.

Thank you for your great work!

[mike2nl]

@psychowood have you searched here about: send UDP packets from javascript

and here: https://stackoverflow.com/questions/13964972/send-udp-packet-from-chrome

and here: https://brightsign.zendesk.com/hc/en-us/community/posts/209965477-Send-UDP-from-a-HTML-interface

My very short google search: https://www.google.com/search?q=send+UDP+packets+from+javascript&rlz=1C1VFKB_enNL911NL911&oq=send+UDP+packets+from+javascript&aqs=chrome..69i57&sourceid=chrome&ie=UTF-8

[psychowood]

@mike2nl Yes I did, and the answer is always the same: you can't, not without experimental stuff (unsupported and deprecated Chrome API for extensions) or a backend proxy (php, in your example).

This is more or less the synthesis:

The 2nd approach (http proxy to udp) could probably be the easiest one to develop in WLED, with the WebRTC one being the "cleanest".

[Aircoookie]

Hi! As I haven't really heard of WebRTC before, I did a quick google and it appears to be more for browser-to-browser communication (e.g. for conferencing apps). What we'd really need is browser-to-server (server being the WLED ESP unit) communication, for which WebSockets appear to be the best asyncronous protocol available. WebSockets is in fact already supported in WLED and I would just need to expand it with a binary format that enables receiving live raw pixel values. Does that sound like a good idea to enable LED streaming from javascript?

[psychowood]

Hi! the main problem could be that websockets are over TCP, thus preserving order (good) but not really allowing packet skipping and potentially adding lag (bad). Theoretically, WebRTC is closer to a real UDP. There's a nice comparison here.

On the other hand, the WebRTC protocol is quite complex to implement and to handle, AFAIK, and this StackOverflow answer affirms that a good websocket is quite usable to near-realtime communications.

Thus, if websockets are already supported, I'd say that the choice is a no-brainer, don't mess with WebRTC :)

[Hokyjack]

The problem I see with WebSockets is that it doesn't allow you broadcast/multicast. Therefore, the WLEDs will not be really synchronize between each other. Another solution for this problem could be to create a simple bridge desktop/phone app, in python or java which would connect via websocket to javascript client and then re-send the message via UDP to WLED devices

[spiro-c]

I just did some doggy POC for sending websocket binary data to wled device https://github.com/spiro-c/WLED/tree/websocket_data it is working good up to 954 pixels here is some demo https://streamable.com/ludxas im sending raw 24bit RGB to wled i have problem when the frame is split into multiple packets after the second frame part for the 955 pixels some of the data is in the second frame and other data is in the third frame ... what can be see on the video

[blazoncek]

@spiro-c WLED, unfortunately, doesn't handle fragmented WS packets. IDK if it is an issue of underlying libraries.

[spiro-c]

@spiro-c WLED, unfortunately, doesn't handle fragmented WS packets. IDK if it is an issue of underlying libraries.

I know is not implemented in wled the ESPAsyncWebServer support fragmented data but with my limited knowledge with buffers and moving shifting data this is best i can do up to 2 packets of data :) according to my test when is send 3072 bytes of data from the JS to wled i received 3 packets in wled so the first packet have length of 1428bytes 476 pixels data second packet have length of 1436 from 477 to 954 pixels plus the Red and Green value from the 955 pixels and the third packet length of 208 blue value from 955 pus rest 69 pixels to 1024 what is visible here

[blazoncek]

@spiro-c I briefly checked the code and on a first look everything is ok. You may add a bit more debugging outputs to see what's happening. I would also recommend to add a header so different payloads can be distinguished in the future.

[spiro-c]

I did little bit more testing and this is what is working for me.... https://github.com/Aircoookie/WLED/compare/0_15...spiro-c:WLED:websocket_data?expand=1 again this is only POC and I this is how much I can do with my limited knowledge in programming If this is useful for some one else to make some better solution or to fully integrated in WLED

https://github.com/Aircoookie/WLED/assets/47688491/62a37142-ac5e-42e1-b42b-eb6f80a124f6

This is the JS bit that is sending the data to WLED it uses the Canvas API most of the code used is from https://github.com/bitluni/PicoWMatrixSceenShare

        function connect() {
            ip = document.querySelector("#ip").value;
            localStorage.setItem('ip', ip);
            socket = new WebSocket(ip);
            socket.onopen = function () {
            };
        }
        function send(val) {
            if (socket.readyState == 1 && socket.bufferedAmount == 0)
                socket.send(val);
        } 
        async function convertCanvas(ctx, xres, yres) {
            //let start = performance.now();
            let imgData = ctx.getImageData(0, 0, xres, yres);
            let pixels = imgData.data;
            //socket.send(Uint8Array.of(0x00));
            var buff = new Uint8Array(xres * yres * 3);
            for (var y = 0; y < yres; y++){
                for (var x = 0; x < xres; x++) {
                    buff[(y * xres + x) * 3 + 0] = pixels[(x + y * xres) * 4 + 0];
                    buff[(y * xres + x) * 3 + 1] = pixels[(x + y * xres) * 4 + 1];
                    buff[(y * xres + x) * 3 + 2] = pixels[(x + y * xres) * 4 + 2];
                }
            }
            socket.send(buff)
            //let end = performance.now();
            //console.log(`Send payload time: ${end - start} ms`);
        }