serial transparent link with websocket

[mkeyno]

hi @Markus , I'm working on the 3d printer board which its serial pins connected to serial pin of ESP module. , in the module couple of html page store in its flash , these file send g-code lines to printer, my plan is to lunch the secure websocket link to printer and send command by its web interface , however I could send the code to printer but its seems the websocket cant handle the respond message of the printer , I find couple of efforts to make wifi-serial bridge but the code in written in lua and I not love lua as much as love @igrr SDK code so any help really appreciated

[Links2004]

do your printer has a binary protocol like repetier host for example then you may need to transfare the data with sendBIN since the sendTXT only allowed utf8 chars (see RFC for more info). how the error exact looking like, to see the code may helps too.

[luc-github]

Hi @Links2004, l am sorry to jump into discussion but I think it better to ask here than https://github.com/esp8266/Arduino/issues/1769 as @mkeyno request is more related to your library and than ESP core itself.

@mkeyno I understand the need of security using SSL and Basic auth, but I do not get how repetier host will handle the websocket authentication, as of today it does not do as far I know, repetier host does not have any authentication between host and printer, when repetier host server has authentication but only for front end, not for communication with printer itself. My understanding is today Marlin / Repetier fw do not support authentication so that is why no host (repetier/pronterface, etc.) have it.

@Links2004 am I wrong ?

Sorry if my question looks stupid but I miss something I think

[mkeyno]

hi @luc-github yes you are right , we don't have any secure link between repetier host server and CNC contrl board and that's why we need put the both feature in ESP module , websocket is good one whereas we could host it in the module flash, also we have enough room to put the @openhardwarecoza in it https://github.com/openhardwarecoza/LaserWeb

[ghost]

Sounds awesome! LaserWeb2 doesn't need the nodejs server anymore, right now i talk via SPJS but if we can have WS on the ESP it will be gold! @raykholo @johnlauer @arthurwolf On 14 Mar 2016 09:06, "mehrdad" notifications@github.com wrote:

hi @luc-github https://github.com/luc-github yes you are right , we don't have any secure link between repetier host server and CNC contrl board and that's why we need put the both feature in ESP module , websocket is good one whereas we could host it in the module flash, also we have enough room to put the @openhardwarecoza https://github.com/openhardwarecoza in it https://github.com/openhardwarecoza/LaserWeb

— Reply to this email directly or view it on GitHub https://github.com/Links2004/arduinoWebSockets/issues/61#issuecomment-196174271 .

[mkeyno]

thanks @openhardwarecoza (peter) to join , yes I think we need this link to enhanced your excellent web interface, local serial connection is like stick us to ground

[ghost]

Did you see my Smoothieboard based work with onboard ESP? (;

https://openhardwarecoza.wordpress.com/2015/12/08/wifi-3d-printing-cnc-laser-with-esp8266-bluetooth-with-hc06/

Keep following me on Google Plus for my most interesting posts though, I hate blogging (; https://plus.google.com/+PetervanderWalt/posts

[luc-github]

@mkeyno sorry again I try to understand your idea, so to sum up your idea, you mean put host software on ESP FW and not using repetier host like application ?

In this case what is the advantage of websocket vs web authentication+https on esp web server ? Websocket is more dedicated to have a client/server communication (like between 2 apps), or I am wrong ?

Because what you describe is everything but transparent bridge, so I am confused

[mkeyno]

hi @luc-github , I think when we use the websocket , we dont need be worry for unexpected other client request and keep the connection alive during the printing just like paring the Bluetooth, and also we could put light webpage such as @openhardwarecoza LaserWeb2 in ESP so need no more third party software such as repetier host, all in one

[mkeyno]

hi @openhardwarecoza , I knew about your board but I think the Due shield is more flexible to make any prototype , and I really appreciated if you could my cnc shield and let me have your comments BR

https://github.com/mkeyno/Due-Shield-CNC-Laser

[luc-github]

@mkeyno so if no repetier host, then websocket communicate with what ? what is for ?

[ghost]

@luc-github - I am jumping into someone elses conversation but what I have a need for is a websocket->serial bridge

Where the application (and for that matter what the application is) is irrelevant for the most part, it could be a web page like Chilipeppr, or a standalone app like LaserWeb.

Adding websocket support allows us to reach the serial bridge, without any "agent" software on the machine. (ie no serialport-json-server or node-serialport needed)

So in my example, where I have http://openhardwarecoza.github.io/LaserWeb2 -> instead of connecting to SPJS which then needs a Pi or PC to bridge to Serial, I could have a ESP with a websocket server, that I can connect to and do a websocket send to: https://github.com/openhardwarecoza/LaserWeb2/blob/gh-pages/js/spjs.js

[ghost]

Websocket of course is a bidirectional method (unlike having to use ajax and poll status updates etc)

The ESP could send out any serial to the websocket, as well as listen for command...

[ghost]

The webpage hosted from the ESP could include same JS and speak back to the ESP over WS - or external web apps.

[mkeyno]

thanks @openhardwarecoza Peter for explain the idea , Dear @luc-github this all in one solution and we could enhance reliability and security if host embedded in the ESP like the websocked webserver

[luc-github]

@openhardwarecoza yes fully understand - your application use websocket to communicate to ESP/printer, that is clear Sorry @mkeyno you confused me using the "All in One" wording, this is not the case, for me "all in one" means no external application for me, not same as what you have in mind, that is why I did not get it.

I also planned to integrated websocket support as alternative of pure TCP connection in my project (https://github.com/luc-github/ESP8266) thanks to the great library of Markus, but I wanted to understand other people needs first, I do not use laser cnc yet (I plan to build one soon) but I guess needs are same as 3D printers. I am clear now thanks a lot

[Links2004]

some notes to the websockets server and ssl. currently there is no ssl tcp server for the ESP (work is ongoing here: https://github.com/esp8266/Arduino/issues/1740) as soon this its do a wss server is possible for the ESP. the Basic auth is possible for the websockets in the same way then its possible for http.

i working on project which does tunnel serial to the browser (remote debugging / router conf) I think it will work in the same way for a 3D printer (have here one may give it a try) but I have currently low free time so no fast progress here.

[mkeyno]

thanks @Links2004 , so you say I have to wait till tunnel serial to the browser be ready ?regards to other notes and codes of @openhardwarecoza and @luc-github is it good option to choose another alternative?

[Links2004]

you can do it in you own way if you like, current code for non blocking bridge (may it helps you)

WebSocketsServer webSocket = WebSocketsServer(81);

#define SEND_SERIAL_TIME (50)

class SerialTerminal {
    public:
        void setup() {
            _lastRX = 0;
            resetBuffer();
            Serial.begin(921600);

        }

        void loop() {
            unsigned long t = millis();
            bool forceSend = false;

            size_t len = (_bufferWritePtr - &_buffer[0]);
            int free = (sizeof(_buffer) - len);

            int available = Serial.available();
            if(available > 0 && free > 0) {
                int readBytes = available;
                if(readBytes > free) {
                    readBytes = free;
                }
                readBytes = Serial.readBytes(_bufferWritePtr, readBytes);
                _bufferWritePtr += readBytes;
                _lastRX = t;
            }

            // check for data in buffer
            len = (_bufferWritePtr - &_buffer[0]);
            if(len >=  sizeof(_buffer)) {
                forceSend = true;
            }
            if(len > (WEBSOCKETS_MAX_HEADER_SIZE + 1)) {
                if(((t - _lastRX) > SEND_SERIAL_TIME) || forceSend) {
                    //Serial1.printf("broadcastBIN forceSend: %d\n", forceSend);
                    webSocket.broadcastBIN(&_buffer[0], (len - WEBSOCKETS_MAX_HEADER_SIZE), true);
                    resetBuffer();
                }
            }
        }

    protected:
        uint8_t _buffer[1460];
        uint8_t * _bufferWritePtr;
        unsigned long _lastRX;

        void resetBuffer() {
            // offset for adding Websocket header
            _bufferWritePtr = &_buffer[WEBSOCKETS_MAX_HEADER_SIZE];
            addChar('T');
        }

        inline void addChar(char c) {
            *_bufferWritePtr = (uint8_t) c; // message type for Webinterface
            _bufferWritePtr++;
        }
};

SerialTerminal term;

void webSocketEvent(uint8_t num, WStype_t type, uint8_t * payload, size_t lenght) {

    switch(type) {
        case WStype_DISCONNECTED:
            Serial1.printf("[%u] Disconnected!\n", num);
            break;
        case WStype_CONNECTED: {
            IPAddress ip = webSocket.remoteIP(num);
            Serial1.printf("[%u] Connected from %d.%d.%d.%d url: %s\n", num, ip[0], ip[1], ip[2], ip[3], payload);

            // send message to client
            webSocket.sendTXT(num, "Connected");
        }
            break;
        case WStype_TEXT:
            Serial1.printf("[%u] get Text: %s\n", num, payload);
            break;

        case WStype_BIN:
            if(lenght > 0) {
                if(payload[0] == 'T') {
                    if(lenght > 1) {
                        Serial.write((const char *) (payload + 1), (lenght - 1));
                       // Serial1.write((const char *) (payload + 1), (lenght - 1));
                    }
                } else {
                    Serial1.printf("[%u] get binary lenght: %u\n", num, lenght);
                    hexdump(payload, lenght);
                }
            }
            break;
    }

}

void setup()
{
    // use Serial 1 for debug out
    Serial1.begin(921600);
    Serial1.setDebugOutput(true);

    Serial1.println();
    Serial1.println();
    Serial1.println();

    for(uint8_t t = 4; t > 0; t--) {
        Serial1.printf("[SETUP] BOOT WAIT %d...\n", t);
        delay(1000);
    }

    Serial1.printf("[SETUP] HEAP: %d\n", ESP.getFreeHeap());

    webSocket.begin();
    webSocket.onEvent(webSocketEvent);

    WiFi.mode(WIFI_AP_STA);
    WiFi.softAP("ESP_SLDT", "yesSecureHere");

    WiFiMulti.addAP("wifi", "yesSecureHere");

    term.setup();

    // disable WiFi sleep for more performance
    WiFi.setSleepMode(WIFI_NONE_SLEEP);
}

void loop()
{
    term.loop();
    webSocket.loop();
    if(WiFiMulti.run() == WL_CONNECTED) {

    }
}

[mkeyno]

hi @openhardwarecoza peter , have you check new codes of Markus on serial bridge

[ghost]

Sorry no, i have been stuck in hardware mode his week.. (: Designed some boards, started building my new pick and place machine ( https://plus.google.com/+PetervanderWalt/posts/Pc1qrv3Yacb) and the Open Source Espresso machine ( https://plus.google.com/+PetervanderWalt/posts/gifGrwqU2UK boiler ) might try over the weekend On 18 Mar 2016 19:09, "mehrdad" notifications@github.com wrote:

hi @openhardwarecoza https://github.com/openhardwarecoza peter , have you check new codes of Markus on serial bridge

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Links2004/arduinoWebSockets/issues/61#issuecomment-198457411

[mkeyno]

pick & place !! wow you are so cool buddy I also have plan to make small pick & place machine such as TVM802A I'll be follow for your new thread and also I really appreciated to have your idea about @Links2004 markus code

[ghost]

Anyone want to get paid to code up a websocket to serial bridge? https://plus.google.com/+PetervanderWalt/posts/BkzJmtaanGc

[Links2004]

websocket to serial code is existing: https://github.com/Links2004/arduinoWebSockets/issues/61#issuecomment-198231422

AP config code is existing too: https://github.com/tzapu/WiFiManager

upload to SD card via serial is possible, no need for dealing with it on the ESP8266 part. but from my experience upload a bigger gcode file via serial is slow like hell. this is why I simple add a RPI to my 3D printer which streams a nice cam Image too ;)

with some hardware hacking its may possible to share the SD card between ESP and Arduino in exclusive way, best i have seen for write to SD from ESP is ~140KB/s.

[ghost]

@Links2004 ok I dug into this today The sketch from https://github.com/Links2004/arduinoWebSockets/issues/61#issuecomment-198231422

• I can send data from serialport to my javascript application, no problem there.
• But sending text from the browser say from Javascript; ws.send("test"), - doesnt show up in the serial output at all?

Digging a little deeper, it seems that the message arrives at type TEXT The sketch does not forward that to the port. Only for type BIN?

[ghost]

Okay, got it mostly working:

/*
 * WebSocketServer.ino
 *
 *  Created on: 22.05.2015
 *
 */

#include <Arduino.h>

#include <ESP8266WiFi.h>

#include <WebSocketsServer.h>
#include <Hash.h>

const char* ssid = "openhardwarecoza";
const char* password = "aabbccddeeff";

WebSocketsServer webSocket = WebSocketsServer(80);

#define SEND_SERIAL_TIME (50)

class SerialTerminal {
    public:
        void setup() {
            _lastRX = 0;
            resetBuffer();
            Serial.begin(115200);

        }

        void loop() {
            unsigned long t = millis();
            bool forceSend = false;

            size_t len = (_bufferWritePtr - &_buffer[0]);
            int free = (sizeof(_buffer) - len);

            int available = Serial.available();
            if(available > 0 && free > 0) {
                int readBytes = available;
                if(readBytes > free) {
                    readBytes = free;
                }
                readBytes = Serial.readBytes(_bufferWritePtr, readBytes);
                _bufferWritePtr += readBytes;
                _lastRX = t;
            }

            // check for data in buffer
            len = (_bufferWritePtr - &_buffer[0]);
            if(len >=  sizeof(_buffer)) {
                forceSend = true;
            }
            if(len > (WEBSOCKETS_MAX_HEADER_SIZE + 1)) {
                if(((t - _lastRX) > SEND_SERIAL_TIME) || forceSend) {
                    //Serial1.printf("broadcastBIN forceSend: %d\n", forceSend);
                    webSocket.broadcastBIN(&_buffer[0], (len - WEBSOCKETS_MAX_HEADER_SIZE), true);
                    resetBuffer();
                }
            }
        }

    protected:
        uint8_t _buffer[1460];
        uint8_t * _bufferWritePtr;
        unsigned long _lastRX;

        void resetBuffer() {
            // offset for adding Websocket header
            _bufferWritePtr = &_buffer[WEBSOCKETS_MAX_HEADER_SIZE];
            // addChar('T');
        }

        inline void addChar(char c) {
            *_bufferWritePtr = (uint8_t) c; // message type for Webinterface
            _bufferWritePtr++;
        }
};

SerialTerminal term;

void webSocketEvent(uint8_t num, WStype_t type, uint8_t * payload, size_t lenght) {

    switch(type) {
        case WStype_DISCONNECTED:
            Serial1.printf("[%u] Disconnected!\n", num);
            break;
        case WStype_CONNECTED: {
            IPAddress ip = webSocket.remoteIP(num);
            Serial1.printf("[%u] Connected from %d.%d.%d.%d url: %s\n", num, ip[0], ip[1], ip[2], ip[3], payload);

            // send message to client
            webSocket.sendTXT(num, "Connected");
        }
            break;
        case WStype_TEXT:
            Serial1.printf("[%u] get Text: %s\n", num, payload);
            if(lenght > 0) {
                Serial.write((const char *) (payload), (lenght));
            }
            Serial.print('\n');
            break;

    }

}

void setup()
{
    // use Serial 1 for debug out
    Serial1.begin(921600);
    Serial1.setDebugOutput(true);

    Serial1.println();
    Serial1.println();
    Serial1.println();

    for(uint8_t t = 4; t > 0; t--) {
        Serial1.printf("[SETUP] BOOT WAIT %d...\n", t);
        delay(1000);
    }

    Serial1.printf("[SETUP] HEAP: %d\n", ESP.getFreeHeap());

    webSocket.begin();
    webSocket.onEvent(webSocketEvent);

    WiFi.mode(WIFI_AP_STA);
    WiFi.begin(ssid, password);
    WiFi.softAP("ESP_SLDT", "yesSecureHere");

    term.setup();

    // disable WiFi sleep for more performance
    WiFi.setSleepMode(WIFI_NONE_SLEEP);
}

void loop()
{
    term.loop();
    webSocket.loop();

}

[Links2004]

I use binary to support Repetier protocol too (which is binary). for gcode Text is fin but your browser will no accept any non visible char´s in text messages.

sending binary example: https://gist.github.com/hagino3000/1447986

[ghost]

Even though using the arraybuffer send as I have in it at the moment, sort of works, I think a ascii implementation would be a little more solid since we have no binary data.

Do you have a ASCII example version?

ws.onmessage = function(e){
    console.log(e.data)
    var data = "";
    if(e.data instanceof ArrayBuffer){
        var bytes = new Uint8Array(e.data);
        for (var i = 0; i < bytes.length; i++) {
           data += String.fromCharCode(bytes[i]);
       }
    } else {
    data = e.data;  // Suspect might be more solid
}

I'd like to receive the data emitted by the serialport, such that I dont need the reassembly of the arraybuffer (I am having some glitches in replies coming back from the controller... )

I suspect this function in your sketch from above is also optimised for Repetier's binary protocol?

void loop() {
            unsigned long t = millis();
            bool forceSend = false;

            size_t len = (_bufferWritePtr - &_buffer[0]);
            int free = (sizeof(_buffer) - len);

            int available = Serial.available();
            if(available > 0 && free > 0) {
                int readBytes = available;
                if(readBytes > free) {
                    readBytes = free;
                }
                readBytes = Serial.readBytes(_bufferWritePtr, readBytes);
                _bufferWritePtr += readBytes;
                _lastRX = t;
            }

            // check for data in buffer
            len = (_bufferWritePtr - &_buffer[0]);
            if(len >=  sizeof(_buffer)) {
                forceSend = true;
            }
            if(len > (WEBSOCKETS_MAX_HEADER_SIZE + 1)) {
                if(((t - _lastRX) > SEND_SERIAL_TIME) || forceSend) {
                    //Serial1.printf("broadcastBIN forceSend: %d\n", forceSend);
                    webSocket.broadcastBIN(&_buffer[0], (len - WEBSOCKETS_MAX_HEADER_SIZE), true);
                    resetBuffer();
                }
            }
        }

Do you have a ASCII example version?

[Links2004]

change webSocket.broadcastBIN to webSocket.broadcastTXT the rest can stay as it is.

Note: you can play with the SEND_SERIAL_TIME define to find the break even point between latency and throughput.

value to low --> to many TCP packages (WS Messages) --> IP Stack overload --> data lost --> print fail too high value --> slows down the data transmit --> slows down the print process

the function is written to handle any kind of data including binary, have used the same code to debug a router remotely ;)

[ghost]

Alright! Thanks! Much better already Now, I have on last requirement in my application (: (I already added you as credit for this - as with these comments you pretty much did all the work https://github.com/openhardwarecoza/LaserWeb3/blob/master/ESP8266%20Wifi%20Bridge/websocketserver.ino/websocketserver.ino.ino#L10 )

In my nodejs based serial to websocket bridge, we get a websocket message for each line from the controller. Acting on these events on the JS side makes things a lot easier

With this sketch, I get a couple lines at a time as a response back in one websocket message I understand the reason, minimise network traffic by filling up a message to the max, then sending it over. But it is causing me some funny issues still

[Links2004]

you can simple force a send when "\n" is found. add a check like:

for(size_t i = WEBSOCKETS_MAX_HEADER_SIZE; i < len; i++) {
    if(_buffer[i] == '\n') {
        forceSend = true;
        break;
    }
}

after

   if(len >=  sizeof(_buffer)) {
           forceSend = true;
    }

but doing the reassembly on the javascript side is a better option from my point of view. at least my gcode interpreter in nodejs does it like this, and it working good and improve the data rate. logic is not that complex.

pseudocode:

var buffer = "";
event.on('data', function(new_text) {
  buffer += new_text;
  var split = buffer.split("\n");
  buffer = split.pop(); //last not fin data back to buffer
  for (var line in split) {
    console.log(line); // trigger line handling event here
  }
});

[ghost]

Alright!!! Now we are solid, and optimised! (:

https://github.com/openhardwarecoza/LaserWeb3/commit/ebd13f0866f4d90e434c39cd5d65c05dfa34ee9f

I really really owe you a couple beers!

@mkeyno - guess we can close this one now?

I'll make a pull request for the total sketch so we can include it in this Library's examples (:

[mkeyno]

good job @openhardwarecoza but as you aware most developer use async approach to make robust connection line, I've used @hallard async bridge for my project , even Luc has revised all his code by async sever, after I've changed it to async approach my websocket service never broke again

[Links2004]

@mkeyno the websocket lib can run sync and async ;) https://github.com/Links2004/arduinoWebSockets/blob/master/src/WebSockets.h#L58-L59

[ghost]

I spent a couple days now trying to get this working 100% solidly:

I still sometimes get

Sent: G1 X65.510 Y55.232 E62.22676 Q: 62814
ok
Sent: G1 X66.118 Y54.893 E62.30940 Q: 62813
ok
Sent: G1 X66.692 Y54.478 E62.39356 Q: 62812
ok
Sent: G1 X67.210 Y53.994 E62.47770 Q: 62811
o2
ok
Sent: G1 X67.662 Y53.456 E62.56121 Q: 62810
 ok

Note that o2 response - looks like something gets mangled

Here's my host https://github.com/openhardwarecoza/CoPrint

Specifically - here's my websocket implementationhttps://github.com/openhardwarecoza/CoPrint/blob/gh-pages/js/esp8266.js

and here's my Arduino sketch for the ESP8266 https://github.com/LaserWeb/LaserWeb3/blob/master/ESP8266%20Wifi%20Bridge/websocketserver.ino/websocketserver.ino.ino

[ghost]

Print via ESP8266 (lots of lost moves)

Print via USB

@luc-github

[ghost]

@raykholo @darklylabs - just FYI above, still a little instability on the websocket side

[Links2004]

after the TCP layer there is no Possibility for bit corruption, I thinks its more a RX problem, on the Serial side.

have you a scope or logic analyzer to check if the baud rates are matching exact? send some bytes on both sides and compare the µs of the bits.

may there is some EMC problem, in one of my projects where high current switching where in involved, it has helped to add a 10k pull up to the serial lines, to get it stable. if its still not stable you can calculate the matching capacitors for the baudrate.

to get the print more stable I recommend to use the checksum feature of gcode. for better tracking for which Gcode the ok is the Line numbers helps too.

[ser007]

Hi @Links2004, thanks for your explanations and codes!

if(len > (WEBSOCKETS_MAX_HEADER_SIZE + 1)) {
    if(((t - _lastRX) > SEND_SERIAL_TIME) || forceSend) {
        //Serial1.printf("broadcastBIN forceSend: %d\n", forceSend);
        webSocket.broadcastTXT(&_buffer[0], (len - WEBSOCKETS_MAX_HEADER_SIZE), true);
        resetBuffer();
    }
}

I'm trying to do a simple validation of the content (string) of & _buffer [0] in the following line:

webSocket.broadcastTXT(&_buffer[0], (len - WEBSOCKETS_MAX_HEADER_SIZE), true);

However, I have no solid knowledge of c / c ++.

I tried something like:

String str = (char*)&_buffer[0];

But I did not succeed :(

Can you help me?

Thank you

[Links2004]

the code use the first bytes to store the webSocket header (bin data) this is done to skip the internal memcopy step that is normally needed inside the Lib. by doing this the memory usage is reduced and its faster.

you can try:

String str = (char*)&_buffer[WEBSOCKETS_MAX_HEADER_SIZE];

when you serial data is ascii it will work fine. may a null (0x00) is needed to mark the end for the String class.