ESP8266 Enhance Performance

[ghost]

Splitting off from #2 - to avoid the discussion drowning in that one

1) I have a rather urgent need to work on this for a non-LW application too 2) This task falls under my new dayjob at OpenBuilds - which means I will have about a week to work on this uninterrupted. Yay 3) If I work on this I may be able to take some of the stress of the other team members

So... Off to start thinking about this. Now that we have comms in the backend for ESP I am considering replacing websocket with a telnet implementation instead. That has proved to work well in a benchtest earlier this week.

Open to thoughts though @cprezzi @DarklyLabs

Pitfalls

I am aware of the conns that we cant then just host LW somewhere and connect to an ESP on the LAN.

But as time has gone on, I honestly believe LW will be only an Electron app. Thats definitely the way to go for distribution, update, code signing, and all the other things we have to think about (speaking of code signing - https://mspoweruser.com/microsoft-just-added-the-best-way-of-preventing-installation-of-bloatware-in-windows-10/ is a particularly worrying thing. We may need to start signing with Authenticode and just get into the windows store if thats where its heading in the future). It also solves some of the cross platform issues. Its not like one can run a laser shop without at least one PC in the shop.... So I dont think ending up with a solution that requires the application to be installed somewhere, is a big issue. server.js can even be run on a SBC (Pi) without the GUI if thats needed

[cprezzi]

I'm not sure yet, if websocket realy is the problem, or just the additional delay the ESP is generating, but you will find out. At least a lower level network protocoll could reduce the performance need on the WLAN<>Serial gateway.

I was already thinking if it's possible to crunch the Websocket<>Serial part of lw.comm-server into an ESP32 directly or at least a C.H.I.P or the new RPi Zero W, so we wouldn't need an additional server for such installations.

You are right, it's enough to publish one electron installer (per OS) that has the app and the server in it. The integrated server doesn't bother much if not used. But I wouldn't drop the option to install the server (without GUI) separately on a machine like an RPi or a spare workshop PC, even if this will probably not be used very often.

By the way: This is like Repetier Host works. All workstations also have a server installed, even if not used (but can be used for "cloud computing").

[ghost]

Agreed. I am working on a --nogui switch in the electron executable (aka server only mode)

Do you want a couple ESP32s? PM me and i'll get a few ordered.

On Feb 28, 2017 6:19 PM, "Claudio Prezzi" notifications@github.com wrote:

I'm not sure yet, if websocket realy is the problem, or just the additional delay the ESP is generating, but you will find out. At least a lower level network protocoll could reduce the performance need on the WLAN<>Serial gateway.

I was already thinking if it's possible to crunch the Websocket<>Serial part of lw.comm-server into an ESP32 directly or at least a C.H.I.P or the new RPi Zero W, so we wouldn't need a server for such installations.

You are right, it's enough to publish one electron installer (per OS) that has the app and the server in it. The integrated server doesn't bother much if not used. But I wouldn't drop the option to install the server (without GUI) separately on a machine like an RPi or a spare workshop PC, even if this will probably not be used very often.

By the way: This is like Repetier Host works. All workstations also have a server installed, even if not used (but can be used for "cloud computing").

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283086699, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr2_zdoOvMY26tHUzPm2HQRp9bZ6NGks5rhEkmgaJpZM4MOhXw .

[cprezzi]

I would still integrate a mini Webpage for mobile / tablet control in the server only version, if possible.

Would be cool if you could organize some ESP32s for me. Sent you a PM.

[ghost]

@openhardwarecoza Can you explain this a little more? We have machines out now which use the ESP setup you helped develop for us. It is working as we would like with our machines appearing on the network and allowing customers to connect directly to them. Why change this now? What will it improve? Will customers with multiple machines be supported? How will we instigate this update to the ESP firmware?

[ghost]

Thats chapter two. Lets see if telnet gives us any improvement first...

For what its worth, OTA update is enabled in the sketch, so updating firmware in the field is possible so dont worry too much

On Feb 28, 2017 11:22 PM, "DarklyLabs" notifications@github.com wrote:

@openhardwarecoza https://github.com/openhardwarecoza Can you explain this a little more? We have machines out now which use the ESP setup you helped develop for us. It is working as we would like with our machines appearing on the network and allowing customers to connect directly to them. Why change this now? What will it improve? Will customers with multiple machines be supported? How will we instigate this update to the ESP firmware?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283164158, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr21TZnw9efyPoQu4mXM3nyw1oqELgks5rhJAQgaJpZM4MOhXw .

[ghost]

@openhardwarecoza Ok, taking a very deep breath! Can you point me to where we can see how to perform an OTA update to the ESP module?

[ghost]

Full details here http://esp8266.github.io/Arduino/versions/2.0.0/doc/ota_updates/ota_updates.html

Basically if you are connected to the same wifi as the module, it pops up on the Port list in Arduino

We can repack the flash tool into a little standalone one or integrated into lw at some point as well so users dont need Arduino

On Mar 1, 2017 12:32 AM, "DarklyLabs" notifications@github.com wrote:

@openhardwarecoza https://github.com/openhardwarecoza Ok, taking a very deep breath! Can you point me to where we can see how to perform an OTA update to the ESP module?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283182809, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr2xOk5ryivxaZt6eIhAbXAeOL6MgYks5rhKCOgaJpZM4MOhXw .

[cprezzi]

Should I activate Telnet in dev_comms? It's already implemented, but never tested, so I disabled it for the moment.

[ghost]

That would certainly help

On Mar 1, 2017 10:57 AM, "Claudio Prezzi" notifications@github.com wrote:

Should I activate Telnet in dev_comms? It's already implemented, but never tested, so I disabled it for the moment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283282099, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr2zPYR8frH10ZXQ_xg-5s6MiPTALcks5rhTLkgaJpZM4MOhXw .

[cprezzi]

Ok, I'll do.

[ghost]

Ok, so I created two very simple nodejs scripts (extracted sections from lw.comms-server) to time the delay between sending a gcode, and receiving back the OK:

telnet

var serialport = require("serialport");
var SerialPort = serialport;
var net = require('net');

var buffer, date, olddate, telnetBuffer;

var ws = net.connect(23, '10.2.0.203');
// ws.binaryType = "arraybuffer";

ws.on('connect', function (e) {
  console.log(e);
  console.log("telnet Connected");
  setTimeout(function (argument) {
    ws.write('G1 X1\n');
  }, 1000);
  setTimeout(function (argument) {
    ws.write('G1 X1\n');
  }, 5000);
});

ws.on('data', function (response) {
    // console.log('...')
                  //var bytes = new Uint8Array(data);
                  for (var i = 0; i < response.length; i++) {
                      if (response[i] != 0x0d) {
                          telnetBuffer += String.fromCharCode(response[i]);
                      }
                  }
                  var responseArray;
                  if (telnetBuffer.substr(-1) === '\n') {
                      responseArray = telnetBuffer.split('\n');
                      telnetBuffer = responseArray.pop();
                  } else {
                      responseArray = telnetBuffer.split('\n');
                      telnetBuffer = '';
                  }
                  var data = '';
                  while (responseArray.length > 0) {
                      data = responseArray.shift();
                      // console.log('Telnet: ' + data, 3);
                      if (data.indexOf('ok') === 0) { // Got an OK so we are clear to send
                        var end = new Date() - date;
                        console.log("time: %dms", end);
                        date = new Date();
                        ws.write('G1 X1\n');
                        console.log(data);
                      }
                  }
              });

websocket

var serialport = require("serialport");
var SerialPort = serialport;
const WebSocket = require('ws');
var buffer, date, olddate;

var ws = new WebSocket('ws://10.1.0.240/');
ws.binaryType = "arraybuffer";

ws.onopen = function(e) {
  console.log(e);
  console.log("ESP8266 Connected");
  setTimeout(function (argument) {
    ws.send('G1 X1\n');
  }, 1000);
  setTimeout(function (argument) {
    ws.send('G1 X1\n');
  }, 2000);
};

ws.onmessage = function(e){

  // console.log(e.data)
  var data = "";
  if(e.data instanceof ArrayBuffer){
    var bytes = new Uint8Array(e.data);
    for (var i = 0; i < bytes.length; i++) {
      data += String.fromCharCode(bytes[i]);
    }
  } else {
    data = e.data;
  }
    buffer += data
    var split = buffer.split("\n");
    buffer = split.pop(); //last not fin data back to buffer
    // console.log(split)
    for (i=0; i< split.length; i++) {
      var response = split[i];
      // console.log(response)
      // trigger line handling event here
      if(response.indexOf("ok") != -1 || response == "start\r" || response.indexOf('<') == 0){
        if (response.indexOf("ok") == 0) { // Got an OK so we are clear to send
          var end = new Date() - date;
          console.log("time: %dms", end);
          date = new Date();
          ws.send('G1 X1\n');
          console.log(response);
          // uploadLine()
        } else if (response.indexOf('<') != -1) {
          console.log(response);
        } else {
          console.log(response)
        }
        blocked = false;
      }
  }
};

And the results are in!

Websocket: 52-57ms Telnet: 5-6ms !!!!

[ghost]

telnet

websocket

[ghost]

Added bonus, this drops a whole extra protocol from our stack, since we can use the same protocol for smoothieware ethernet, Smoothieware wifi, grbl wifi and theoretically also tinyg wifi

[cprezzi]

Cool. That sounds very promissing!

[ghost]

I was positively surprised too! Didnt realize that the extra headers used to wrap the packet make such a huge time difference

[cprezzi]

When I implemented the Telnet part, I tried a Telnet library first, but I realised that it is just overhead and decided to use the low level net lib.

[cprezzi]

This means I have to extend the Telnet part in lw.comm-server to support multiple firmwares (like with USB) and not only smoothie, but that's not a big deal ;)

[ghost]

Agreed. Net for the win.

Speaking of, node-serialport, unrelated of course. But github.com/andrewhodel at some point mentioned he wanted to replace serialport with fs and write to /dev/ttyusb for example. nix os's being cool enough that ports are just virtual files. Sadly doesnt work for windows of course so that halted. But same argument, skip the library and go lowlevel

On Mar 2, 2017 11:50 AM, "Claudio Prezzi" notifications@github.com wrote:

When I implemented the Telnet part, I tried a Telnet library first, but I realised that it is just overhead and decided to use the low level net lib.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283607195, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr2xK-MP6pbAQoTTsd2GImot8zL-Lzks5rhpDmgaJpZM4MOhXw .

[ghost]

Sorry (: - looks like the juice is worth the squeeze though (:

On Mar 2, 2017 11:52 AM, "Claudio Prezzi" notifications@github.com wrote:

This means I have to extend the Telnet part in lw.comm-server to support multiple firmwares (like USB) and not only smoothie, but that's not a big deal ;)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283607847, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr21YpvgVCIlSiaBp1dkhquvt14J0Vks5rhpF6gaJpZM4MOhXw .

[cprezzi]

Yep. I like low level anyway. Less dependency to libraries that I can't control ;) That's a bit of a concern with LW4. We just have too many dependencies.

[ghost]

I hear you. 600mb worth of libraries to build a 8mb app. Though i will admit that its more stable the lw3 though hehe

On Mar 2, 2017 12:00 PM, "Claudio Prezzi" notifications@github.com wrote:

Yep. I like low level anyway. Less dependency to libraries that I can't control ;) That's a bit of a concern with LW4. We just have too many dependencies.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LaserWeb/lw.comm-server/issues/10#issuecomment-283609634, or mute the thread https://github.com/notifications/unsubscribe-auth/AHVr23AjzPAGpAeIbZVgs7LvASvcDlcpks5rhpNMgaJpZM4MOhXw .

[ghost]

Just checking in to see if there has been any progress made with the Telnet approach or improving the current server performance over Wifi?