Again: TCP server performance completely unuseful

[nouser2013]

Hi, I'm sorry to bring this up once more as it has been referenced several times now. But still, I cannot get a TCP server to put more than one packet in the send queue due to incorrect handling of the delayed ACK. Looking at the headers, there is the TCP_MSS set to 1460 and the TCP_WND four times that value. Hence, if the same thing is in the liblwip.a (and negotiation works correctly), we shouldn't have a problem.

What bothers me in particular is the following video https://www.youtube.com/watch?v=8ISbmQTbjDI where this guy sends websocket replies back to the client at about 220Hz. And they carry arrays of data. To be fair, he coded this with the FreeRTOS version without arduino intermediary layer.

BUT: why can't we have this?

WiFiServer tcpTelnetServer(23);
WiFiClient tcpTelnetClient; // Only one client at a time a.t.m.!

void setup() {
  tcpTelnetServer.begin();
}

void loop() {
  // HANDLE TCP TELNET SERVER
  if (!tcpTelnetClient.connected()) {
    tcpTelnetClient = tcpTelnetServer.available();
    if (tcpTelnetClient.connected()) {
      tcpTelnetClient.setNoDelay(true);
      Serial.println(" DBG: New TCP client connected...");
    }
  } else {
    // remember: Telnet sends \r or \r\n, NOT \n!
    if (tcpTelnetClient.available()) {
      char stringBuffer[300];
      tcpTelnetClient.readBytesUntil('\r', stringBuffer, sizeof(stringBuffer)-1);
      tcpTelnetClient.flush();
      // Process command...
      for (int i=0; i<6; i++) {
        memset(stringBuffer, 't', 60);
        stringBuffer[60] = 0; // Zeroterminate manually!
        Serial.printf("Before (%d): %d\n", i, micros());
        tcpTelnetClient.print(stringBuffer);
        Serial.printf("Middle (%d): %d\n", i, micros());
        tcpTelnetClient.write((uint8_t *)stringBuffer, strlen(stringBuffer));
        Serial.printf("After  (%d): %d\n\n", i, micros());
      }
    }
  }
}

this results in

Before (0): 46582971
Middle (0): 46785297
After  (0): 46984893

Before (1): 46985136
Middle (1): 47393106
After  (1): 47803031

Before (2): 47803275
Middle (2): 48212294
After  (2): 48621882

Before (3): 48622126
Middle (3): 48827013
After  (3): 49236314

Before (4): 49236558
Middle (4): 49647626
After  (4): 49854952

Before (5): 49855141
Middle (5): 50054827
After  (5): 50254701

*edit: MWE added..

[nouser2013]

Well, after a lot of try and error I seem to have found a workaround. It partly was not my idea but I also don't remember where I read the hint. Here goes:

--- a/libraries/ESP8266WiFi/src/include/ClientContext.h
+++ b/libraries/ESP8266WiFi/src/include/ClientContext.h
@@ -233,7 +233,8 @@ class ClientContext {
             DEBUGV(":wr\r\n");
             tcp_output( _pcb );
             _send_waiting = true;
-            delay(5000); // max send timeout
+            delay(5); // max send timeout
             _send_waiting = false;
             DEBUGV(":ww\r\n");
             return will_send - _size_sent;

I have no idea what this delay is for and if other parts rely on the _send_waiting variable, but when reducing this to 5ms, everything works very nice.

I still lost some packets, particularly right after connection establishment, therefore I also increased the TCP send buffer here:

--- a/libraries/ESP8266WiFi/src/include/lwipopts.h
+++ b/libraries/ESP8266WiFi/src/include/lwipopts.h
@@ -943,7 +943,8 @@
  * TCP_SND_BUF: TCP sender buffer space (bytes).
  */
 #ifndef TCP_SND_BUF
-#define TCP_SND_BUF                     2 * TCP_MSS
+#define TCP_SND_BUF                     4 * TCP_MSS
 #endif

Well, hope this helps others as well. And again, maybe somebody could elaborate on what these 5000ms delay were for. Cheers.

[drmpf]

_send_waiting was my suggestion. Waiting on ivan, markus etc, to review the idea for inclusion. (https://github.com/esp8266/Arduino/issues/922)

In the mean time, a complete library pfodESP2866WiFi with examples is available from http://www.forward.com.au/pfod/pfodParserLibraries/index.html This is a modified version of the ESP8266 V2.0.0 with files renamed so you can install it with the standard code without conflicts. The modified library does a few things i) buffers small sends for upto 10mS to reduce the number of short packets ii) does not block if the packet can be sent immediately iii) adds a method isSendWaiting() so you can check if a call to TCP send will block.

[drmpf]

maybe somebody could elaborate on what these 5000ms delay were for. The 5 sec delay was there to prevent complete lockup if the packet was never responded to. In my replacement library there is a loop above which waits for _send_waiting to become false. It times out after 5 secs and returns -1 for bytes sent to indicate an error trying to send the data

 size_t counter = 0;
      DEBUGV("clientContext write isSendingWaiting: %d",_send_waiting);
      while (isSendWaiting() && (counter < 5000) ) {
        delay(1); // block here waiting for last send to complete
        counter++;
      }
      if (counter >= 5000) {
        DEBUGV("counter timed out!");
        // timed out waiting for last send to complete
        _send_waiting = false;
        return (size_t)-1; // abort connection
      }

[Links2004]

the arduino API is sync so if you write you get a return how many bytes are written. the delay are a timeout in case the write failed and the error handler is not triggert, normally in a good case it skip the delay and only need ~10ms. https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/include/ClientContext.h#L247 or on error: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/src/include/ClientContext.h#L301

will add some more debug there, hope this help to find the reason for the delay problem.

[Links2004]

please use my debug branch and enable debug for Core: https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/debugging.md#Usage and post the result here. good example:

:wr  
:sent 2 left: 0  
:ww 2 us: 1287  

or

:wr  
:sent 69 left: 0  
:ww 69 us: 31096  

[nouser2013]

@drmpf: Ah thanks, I remember now, yes that's the post I read about the delay. I tried your code and while the write() returns almost immediately and sends packets as required only, there was still the issue that under Windows new input sent to the ESP would not be processed right away. In short:

1. connect
2. receive weilcome message
3. send first command
4. send second command right after <= processed only after the infamous 200ms.

Therefore I didn't follow this further.

@Links2004: could you specify exactly what you need? My git and the web interface on github does not show me a branch "debug", I only have master and gh-pages. I do see the debug options in the IDE, several of them with "core", which one should I select?

[Links2004]

its not in the esp8266 git, its a branch in my clone: https://github.com/Links2004/Arduino/tree/debug any will do but just Core is best we not need the other messages.

[nouser2013]

@Links2004, I did as you asked and had a code send several small texts to an open TCP client. This is the send-code:

char outBuffer[1000];
WiFiClient tcpClient;
//[...]
strcpy_P( outBuffer, PSTR("my numerous test strings""\n") );
tcpClient->write(outBuffer, strlen(outBuffer))

And on the serial port I had a small python script add time stamps and otherwise print the output:

 Serial Debug Catcher
 ====================

 2016-01-17 20:01:35.413000 << INFO: Setup GPIO...
 2016-01-17 20:01:35.413000 << INFO: Setup file system (will take a few minutes on first boot!)...
 2016-01-17 20:01:35.491000 << INFO: Setup WiFi...
 2016-01-17 20:01:35.491000 <<  DBG: Config file 'config.txt' found. Parsing...
 2016-01-17 20:01:35.491000 <<  DBG: MODE='CLIENT'
 2016-01-17 20:01:35.491000 <<  DBG: SSID='***'
 2016-01-17 20:01:35.491000 <<  DBG:  PWD='***'
 2016-01-17 20:01:35.491000 <<  DBG: Setting WiFi-STATION mode...
 2016-01-17 20:01:39.287000 << .............................
 2016-01-17 20:01:39.302000 <<  DBG: ...conntected!
 2016-01-17 20:01:39.302000 <<  DBG: IP Address: 192.168.1.208
 2016-01-17 20:01:41.317000 << INFO: Setup UDP DirectControl on port 81...
 2016-01-17 20:01:41.317000 << INFO: Setup TCP telnet-similar on port 23...
 2016-01-17 20:01:41.332000 << INFO: Setup webserver on port 80...
 2016-01-17 20:01:41.332000 <<  DBG: Testing for valid animations...
 2016-01-17 20:01:42.315000 << INFO: Init complete.
 2016-01-17 20:01:48.477000 << pm open,type:2 0
 2016-01-17 20:01:51.443000 << WS:ac
 2016-01-17 20:01:51.443000 << :ref 1
 2016-01-17 20:01:51.443000 << WS:av
 2016-01-17 20:01:51.443000 << :ref 2
 2016-01-17 20:01:51.443000 << :ur 2
 2016-01-17 20:01:51.443000 << :ref 2
 2016-01-17 20:01:51.443000 << :ur 2
 2016-01-17 20:01:51.443000 <<  DBG: New TCP client connected...
 2016-01-17 20:01:51.443000 << :wr _size_sent: 16724272
 2016-01-17 20:01:51.443000 << :wr
 2016-01-17 20:01:51.833000 << :sent 51 left: 0
 2016-01-17 20:01:51.833000 << :ww 51 us: 402850
 2016-01-17 20:01:51.833000 << :wr
 2016-01-17 20:01:52.239000 << :sent 19 left: 0
 2016-01-17 20:01:52.239000 << :ww 19 us: 410805
 2016-01-17 20:01:52.239000 << :wr
 2016-01-17 20:01:52.457000 << :sent 19 left: 0
 2016-01-17 20:01:52.457000 << :ww 19 us: 203844
 2016-01-17 20:01:52.457000 << :wr
 2016-01-17 20:01:52.847000 << :sent 40 left: 0
 2016-01-17 20:01:52.847000 << :ww 40 us: 406965
 2016-01-17 20:01:52.847000 << :wr
 2016-01-17 20:01:53.268000 << :sent 44 left: 0
 2016-01-17 20:01:53.268000 << :ww 44 us: 408822
 2016-01-17 20:01:53.268000 << :wr
 2016-01-17 20:01:53.658000 << :sent 53 left: 0
 2016-01-17 20:01:53.674000 << :ww 53 us: 408874
 2016-01-17 20:01:53.690000 << :wr
 2016-01-17 20:01:54.080000 << :sent 53 left: 0
 2016-01-17 20:01:54.080000 << :ww 53 us: 408707
 2016-01-17 20:01:54.080000 << :wr
 2016-01-17 20:01:54.298000 << :sent 69 left: 0
 2016-01-17 20:01:54.298000 << :ww 69 us: 218329
 2016-01-17 20:01:54.314000 << :wr
 2016-01-17 20:01:54.516000 << :sent 40 left: 0
 2016-01-17 20:01:54.516000 << :ww 40 us: 202809
 2016-01-17 20:01:54.516000 << :wr
 2016-01-17 20:01:54.719000 << :sent 25 left: 0
 2016-01-17 20:01:54.719000 << :ww 25 us: 218235
 2016-01-17 20:01:54.719000 << :wr
 2016-01-17 20:01:54.938000 << :sent 60 left: 0
 2016-01-17 20:01:54.953000 << :ww 60 us: 218791
 2016-01-17 20:01:54.953000 << :wr
 2016-01-17 20:01:55.172000 << :sent 67 left: 0
 2016-01-17 20:01:55.172000 << :ww 67 us: 215937
 2016-01-17 20:01:55.172000 << :wr
 2016-01-17 20:01:55.359000 << :sent 46 left: 0
 2016-01-17 20:01:55.359000 << :ww 46 us: 202498
 2016-01-17 20:01:55.359000 << :wr
 2016-01-17 20:01:55.577000 << :sent 46 left: 0
 2016-01-17 20:01:55.593000 << :ww 46 us: 217536
 2016-01-17 20:01:55.593000 << :wr
 2016-01-17 20:01:55.796000 << :sent 38 left: 0
 2016-01-17 20:01:55.796000 << :ww 38 us: 203567
 2016-01-17 20:01:55.796000 << :wr
 2016-01-17 20:01:55.998000 << :sent 34 left: 0
 2016-01-17 20:01:55.998000 << :ww 34 us: 217325
 2016-01-17 20:01:55.998000 << :wr
 2016-01-17 20:01:56.217000 << :sent 31 left: 0
 2016-01-17 20:01:56.232000 << :ww 31 us: 217108
 2016-01-17 20:02:02.177000 << :rn 1
 2016-01-17 20:02:02.177000 << :c0 1, 1
 2016-01-17 20:02:02.177000 << :rn 2
 2016-01-17 20:02:02.177000 << :wr
 2016-01-17 20:02:02.473000 << :sent 51 left: 0
 2016-01-17 20:02:02.473000 << :ww 51 us: 306281
 2016-01-17 20:02:02.473000 << :wr
 2016-01-17 20:02:02.895000 << :sent 19 left: 0
 2016-01-17 20:02:02.895000 << :ww 19 us: 406999
 2016-01-17 20:02:02.895000 << :wr
 2016-01-17 20:02:03.287000 << :sent 19 left: 0
 2016-01-17 20:02:03.302000 << :ww 19 us: 408797
 2016-01-17 20:02:03.302000 << :wr
 2016-01-17 20:02:03.708000 << :sent 40 left: 0
 2016-01-17 20:02:03.708000 << :ww 40 us: 408813
 2016-01-17 20:02:03.708000 << :wr
 2016-01-17 20:02:04.129000 << :sent 44 left: 0
 2016-01-17 20:02:04.129000 << :ww 44 us: 408841
 2016-01-17 20:02:04.129000 << :wr
 2016-01-17 20:02:04.519000 << :sent 53 left: 0
 2016-01-17 20:02:04.519000 << :ww 53 us: 409111
 2016-01-17 20:02:04.519000 << :wr
 2016-01-17 20:02:04.753000 << :sent 53 left: 0
 2016-01-17 20:02:04.753000 << :ww 53 us: 212658
 2016-01-17 20:02:04.753000 << :wr
 2016-01-17 20:02:04.956000 << :sent 69 left: 0
 2016-01-17 20:02:04.956000 << :ww 69 us: 217837
 2016-01-17 20:02:04.956000 << :wr
 2016-01-17 20:02:05.159000 << :sent 40 left: 0
 2016-01-17 20:02:05.159000 << :ww 40 us: 202267
 2016-01-17 20:02:05.159000 << :wr
 2016-01-17 20:02:05.377000 << :sent 25 left: 0
 2016-01-17 20:02:05.377000 << :ww 25 us: 202136
 2016-01-17 20:02:05.377000 << :wr
 2016-01-17 20:02:05.580000 << :sent 60 left: 0
 2016-01-17 20:02:05.580000 << :ww 60 us: 217916
 2016-01-17 20:02:05.580000 << :wr
 2016-01-17 20:02:05.798000 << :sent 67 left: 0
 2016-01-17 20:02:05.798000 << :ww 67 us: 217839
 2016-01-17 20:02:05.798000 << :wr
 2016-01-17 20:02:06.017000 << :sent 46 left: 0
 2016-01-17 20:02:06.017000 << :ww 46 us: 202262
 2016-01-17 20:02:06.017000 << :wr
 2016-01-17 20:02:06.219000 << :sent 46 left: 0
 2016-01-17 20:02:06.219000 << :ww 46 us: 217707
 2016-01-17 20:02:06.219000 << :wr
 2016-01-17 20:02:06.440000 << :sent 38 left: 0
 2016-01-17 20:02:06.440000 << :ww 38 us: 219920
 2016-01-17 20:02:06.440000 << :wr
 2016-01-17 20:02:06.658000 << :sent 34 left: 0
 2016-01-17 20:02:06.674000 << :ww 34 us: 217898
 2016-01-17 20:02:06.674000 << :wr
 2016-01-17 20:02:06.892000 << :sent 31 left: 0
 2016-01-17 20:02:06.892000 << :ww 31 us: 219665
 2016-01-17 20:02:08.140000 << :rn 1
 2016-01-17 20:02:08.156000 << :c0 1, 1
 2016-01-17 20:02:08.156000 << :rn 2
 2016-01-17 20:02:08.156000 << :wr
 2016-01-17 20:02:08.361000 << :sent 33 left: 0
 2016-01-17 20:02:08.361000 << :ww 33 us: 215674
 2016-01-17 20:02:08.361000 << :wr
 2016-01-17 20:02:08.579000 << :sent 31 left: 0
 2016-01-17 20:02:08.579000 << :ww 31 us: 220161
 2016-01-17 20:02:08.595000 << :wr
 2016-01-17 20:02:08.797000 << :sent 27 left: 0
 2016-01-17 20:02:08.797000 << :ww 27 us: 199579
 2016-01-17 20:02:08.797000 << :wr
 2016-01-17 20:02:09.018000 << :sent 21 left: 0
 2016-01-17 20:02:09.018000 << :ww 21 us: 222375
 2016-01-17 20:02:09.018000 << :wr
 2016-01-17 20:02:09.221000 << :sent 21 left: 0
 2016-01-17 20:02:09.221000 << :ww 21 us: 214396
 2016-01-17 20:02:09.221000 << :wr
 2016-01-17 20:02:09.455000 << :sent 26 left: 0
 2016-01-17 20:02:09.455000 << :ww 26 us: 217401
 2016-01-17 20:02:09.455000 << :wr
 2016-01-17 20:02:09.658000 << :sent 25 left: 0
 2016-01-17 20:02:09.658000 << :ww 25 us: 217586
 2016-01-17 20:02:09.673000 << :wr
 2016-01-17 20:02:09.860000 << :sent 25 left: 0
 2016-01-17 20:02:09.860000 << :ww 25 us: 203084
 2016-01-17 20:02:09.860000 << :wr
 2016-01-17 20:02:10.094000 << :sent 25 left: 0
 2016-01-17 20:02:10.094000 << :ww 25 us: 218079
 2016-01-17 20:02:10.094000 << :wr
 2016-01-17 20:02:10.315000 << :sent 15 left: 0
 2016-01-17 20:02:10.315000 << :ww 15 us: 218461
 2016-01-17 20:02:10.315000 << :wr
 2016-01-17 20:02:10.533000 << :sent 28 left: 0
 2016-01-17 20:02:10.533000 << :ww 28 us: 216246
 2016-01-17 20:02:10.533000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.533000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.533000 << :wr
 2016-01-17 20:02:10.752000 << :sent 28 left: 0
 2016-01-17 20:02:10.752000 << :ww 28 us: 216392
 2016-01-17 20:02:10.752000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.752000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.752000 << :wr
 2016-01-17 20:02:10.954000 << :sent 28 left: 0
 2016-01-17 20:02:10.954000 << :ww 28 us: 201061
 2016-01-17 20:02:10.954000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.954000 << SPIFFS_close: fd=1
 2016-01-17 20:02:10.954000 << :wr
 2016-01-17 20:02:11.173000 << :sent 28 left: 0
 2016-01-17 20:02:11.173000 << :ww 28 us: 216638
 2016-01-17 20:02:11.173000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.173000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.173000 << :wr
 2016-01-17 20:02:11.376000 << :sent 27 left: 0
 2016-01-17 20:02:11.376000 << :ww 27 us: 200537
 2016-01-17 20:02:11.376000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.376000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.376000 << :wr
 2016-01-17 20:02:11.594000 << :sent 29 left: 0
 2016-01-17 20:02:11.594000 << :ww 29 us: 216163
 2016-01-17 20:02:11.594000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.594000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.594000 << :wr
 2016-01-17 20:02:11.812000 << :sent 21 left: 0
 2016-01-17 20:02:11.812000 << :ww 21 us: 216704
 2016-01-17 20:02:11.812000 << SPIFFS_close: fd=1
 2016-01-17 20:02:11.812000 << SPIFFS_close: fd=1
 2016-01-17 20:02:15.166000 << :rn 2
 2016-01-17 20:02:15.182000 << :c0 1, 2
 2016-01-17 20:02:15.182000 << :rn 2
 2016-01-17 20:02:15.182000 << :wr
 2016-01-17 20:02:15.400000 << :sent 33 left: 0
 2016-01-17 20:02:15.400000 << :ww 33 us: 203567
 2016-01-17 20:02:17.990000 << :rn 1
 2016-01-17 20:02:17.990000 << :c0 1, 1
 2016-01-17 20:02:17.990000 << :rn 2
 2016-01-17 20:02:17.990000 <<  DBG: TCP client disconnected...
 2016-01-17 20:02:17.990000 << :wr
 2016-01-17 20:02:18.255000 << :sent 10 left: 0
 2016-01-17 20:02:18.255000 << :ww 10 us: 274648
 2016-01-17 20:02:18.255000 << :ur 1
 2016-01-17 20:02:18.255000 << :close
 2016-01-17 20:02:18.255000 << WS:dis
 2016-01-17 20:02:18.255000 << :del

This is the python script:

import serial
from datetime import datetime
import io
import time

print " Serial Debug Catcher"
print " ===================="
print ""

ser    = serial.Serial(port="COM9:", baudrate=115200, bytesize=serial.EIGHTBITS, parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE, timeout=1)
ser_io = io.TextIOWrapper(io.BufferedRWPair(ser, ser, 1), newline = '\n', line_buffering = True)

while 1:
    reply = ""
    try:
        reply = ser_io.readline().rstrip()
    except:
        reply = ""
    if not reply: continue
    try:
        print " %s << %s" %(datetime.now(), reply)
    except:
        continue

I hope you can use this. Let me know if I can be of more help...

[Links2004]

the log show no error in the TCP level of the ESP. may the ack for the data from the Opponent site is delayed. can you check with wire shark if the TCP ACK has a delay?

[nouser2013]

Oh, I apologise, I thought this was clear from @drmpf 's post. Yes, the ACK is delayed under Windows:

BUT: TCP should manage Window size automatically. For some reason Windows waits for more packets to arrive and only ACKs the single one transmitted after the 200ms timeout. And ESP826 does not send more than one packet, because it didn't receive an ACK for the previous one yet.

Thus I see the error with the ESP code which should send more packets, so that the receiver (here the windows PC) can fill its RX buffer and ACK everything in it with just one ACK per bunch, not one ACK per packet.

Removing the _send_wait delay "forces" the ESP to run into a timeout very quickly and send the next packet "no matter what". Which occasionally results in packet loss even under TCP (Web browser error ERR_CONTENT_LENGTH_MISMATCH). Thus I also increased the send-buffer so that this does not occur.

Lastly, it still remains open why ESP does not send the negotiated number (TCP_WND[_SIZE]/TCP_MSS) of packets (which Windows is expecting), but only one before waiting for an ACK.

If I can help track this down, please tell me how...

[Links2004]

the packet loss can be prevented (simple increasing buffer and hoping is no good idea). the TCP need a other "TX mode" to behave like the serial, which only blocks when buffer full. but the default mode need to be block until ACK, or we break many existing code. but we need a good error handling for this new mode.

[drmpf]

My suggested mods add buffering and works with existing code, the use of isSendWaiting() is optional, as is the buffering, Check out the examples included with my sample library

[nouser2013]

Hmm, perhaps the issue really is when socket.send() should return to user code. Please find this good discussion on the matter. I would agree that user code can fill the kernel buffer up to its max by calling send() and only receives an error similar to EWOULDBLOCK (or actually blocks) if the kernel memory is full.

This is more so as other operating systems than Windows may feature a delayed ACK TCP stack functionality and ESP should cope with that. Also bear in mind that not all TCP applications require replies from the clients; HTTP downloads for instance. Therefore the ESP should send packets up to the receivers window size as advertised in the initial SYN message and each response packet. I don't see a reason why we cannot?

So, if we need to notify user code of any send error, I'd suggest registering a callback for that: In case the kernel cannot transmit data in its buffer the callback function is executed with the appropriate error code. Resynching data stream is user code responsibility. Other possibility would be to register a callback which send()s next data chunk to kernel memory, as in the SDK.

This would break Arduino functionality-style only partly and still cope with delayed ACKs on any operating system (i.m.h.o.).

[Links2004]

my plan is to add a option to the to allow, filling the internal buffer up, and only block when the buffer is full, same concept like the Serial is operating on write/print.

but the default option will be the behavior like it is today (we brake to many thinks if we change it).

[igrr]

Yeah, changing the current behavior will break existing code which assumes that client.write waits for the data to be transmitted and ACKed. Best option would be to introduce a different library with posix-like API, and with the semantics of read/write which you expect.

On Mon, Jan 18, 2016, 19:44 Markus notifications@github.com wrote:

my plan is to add a option to the to allow, filling the internal buffer up, and only block when the buffer is full, same concept like the Serial is operating on write/print.

but the default option will be the behavior like it is today (we brake to many thinks if we change it).

— Reply to this email directly or view it on GitHub https://github.com/esp8266/Arduino/issues/1430#issuecomment-172582755.

[Links2004]

for full async TCP check out ESPAsyncTCP from @me-no-dev https://github.com/me-no-dev/ESPAsyncTCP

[copterfan20001]

I understand and appreciate this. But this concerns just the behaviour of the send() function. The ESP8266 kernel, on the other hand, must somehow cope with delayed ACKs of the recipient or otherwise this will be for nothing.

Meaning ESP must honour the size of the recipient's RWin and keep sending its own send-buffer until:

1. own (ESP) buffer sent completely
2. RWin of recipient full (or would be full with next packet)

Remember, it's a dynamic process, the recipient will advertise a diminishing space in RX buffer with its own ACK.

And a dirty hack to keep existing fucntionality and cope with delayed ACKs: if message_size>1 then send two TCP packets =)

[me-no-dev]

@copterfan20001 check the async server that @Links2004 linked you with. I send two packets on response if there is content length. One with the headers and one with the data (as far as I remember). Also since it's async, it does not block anything and does it's job in background. The client is also properly disconnected and freed. Requires another library that you will find in my github (ESPAsyncTCP). I also support "ExpectContinue" header where the sender expects response before sending the rest of the request.

Please let us know if this is of any help and if so I'll hurry up the merge to this repo.

[copterfan20001]

@me-no-dev oooooooooooh, very nice job. I'll check it out this evening (in a few hours) and let you know :) But from the looks this is exactly what I've been looking for. Is it possible at all to integrate into the Arduino lib without breaking current functionality?

[me-no-dev]

just put the downloaded libs in your libraries folder and they will work. As for compatibility with the current server, I hope to be able to make it as much the same as I can without breaking anything, but we will keep both :) there are too many examples and whatnot that run on it.

[nouser2013]

so, at home now. This may sound like a total n00b question, but how exactly would I go about using the files? I copied "as-is" to library folder, included the *.h without error, and would like to register my first onClient callback....... How do I do that? Thanks.

[me-no-dev]

Are you talking about the AsyncWebServer? it has an example that shows 2 main features. How to add the default handlers (the same way as in ESP8266WebServer but slightly different arguments). In those handlers all available info is printed for the request but you can modify as needed. The other feature is the top part of the example and it shows a full file web server with the same capabilities as the example FSBrowser for the regular server but implemented as a sort of plugin. The api it uses to talk to the client is the same as in the callbacks (they are called from similar class). Overall if you read the example it should be self explanatory :) i hope

[nouser2013]

Nah, wanted to try simple TCP server first, so went to https://github.com/me-no-dev/ESPAsyncTCP and downloaded everything and put that into the library folder. Then I included the *.h, work fine. But now how would I register the onClinet callback, actually get the new client object within that callback and register the onReceive function for that client? Would you have a small telnet-server like example?

[me-no-dev]

like this? https://gist.github.com/me-no-dev/116e417ea6a3bbc98b08

[nouser2013]

oooooooooooh, marvellous! I particularly like your (imo) clean coding style. How can I get you a beer???? 2x thumbs up from my side, couldn't test the error function though, the ACKs were sent cleanly.

[me-no-dev]

Knowing that it helped and it is working is enough for me :) if it's not for you, then my paypal is ficeto@ficeto.com :P drop me an issue report if you find any bugs or have recomendations. You can take a look at the WebServer class for keeping and cleaning multiple clients.

[nouser2013]

Hi, hope you've had a cold one already, needed to get paypal first :) One micro question/suggestion. With HardwareSerial and the WiFiClient object I can do something like:

// "Serial" is built-in
WiFiClient tcpClient;
AsyncClient tcpAsyncClient; // Sic! no pointer.
Print *outChannel;

void setup() { /*...*/ };
void loop() {
  outChannel = &Serial;
  outChannel->write("Hello SERIAL"); // goes to Serial
  outChannel = &tcpClient;
  outChannel->write("Hello TCPCLIENT"); // goes to established WiFiClient connection
  outChannel = &tcpAsyncClient; // compile error here
  outChannel->write("Hello TCPASYNCCLIENT"); // should go to established ASYNC connection
}

Trying the same with your client results in a "cannot convert Print to AsyncClient" compile error, most likely I'm doing something wrong. The write() function signature is the same afaik, but still it doesn't work. Thought it may be the function pointers, but I don't have any idea how to do function pointers to class member functions. Would you have a suggestion for that as well?

I'm using this to have a universal command processor btw, i don't care if a command comes in via Serial, Telnet, HTTP, UDP, or different, I just need the buffer and its length. Before calling the command processor, I'd point the outChannel to the correct function container so the command processor output goes to the correct channel automatically. I avoid a lot of if's in the processor that way. Perhaps it would be cleaner to function-point to the respective write() function directly? Hope that's a c way of doing stuff...

[me-no-dev]

The Async Client is meant to be a bit different :) it's not exactly a Stream like what you expect in the code above. You can wrap it inside a simple class though that extends Print and implement write(byte) method that will make your print work, but beware! writing 1 byte to the network will send a packet for that byte and will expect ack before can send another byte. That is why it's not a stream :) The more you get to send at once (up to 1460 bytes), the better and more speed you will get. If you look at the example I gave you, when I'm reading the serial I actually give it a chance to get some bytes if it's empty so I can send more of the incoming message at once. You would need something like that implemented so not to send a packet with each byte :)

[me-no-dev]

as an idea, if you are intending to use it to pass commands or messages terminated by new line, in that write(byte) method, you can check and see if the byte is a new line character and send the buffered string (most terminals and shells work that way). If you think it's above your ability to write such thing, let me know and I'll wrap something for you.

[drmpf]

You could also just wait a little to see if more data is written, before sending a partial packet, which is what pfodESP2866WiFi does when you uses its buffering

[me-no-dev]

here is an example if a printing Async Client https://gist.github.com/me-no-dev/c86a219fc2f2346c18d0 What it does is fill the buffer until 1460 bytes (max packet) are reached and sends it, unless the client's regular poll callback passes by to send what is currently available in the buffer (every 125ms or so).

[me-no-dev]

and here is another go that you might like better (printer is not a reference) https://gist.github.com/me-no-dev/8e03b214d2c6061bd8b6

[nouser2013]

@me-no-dev thanks very much again, that's a lot of effort you put in there. I apologize again for not being very clear. The stream functionality of those various classes is not so important. It's more about c++/c semantics on how to correctly "function-point". So, I'll always have a uint8_t buf[] with a known length uint16_t len that I want to write(const uint8_t *, uint16_t) via a specific void *outChannel. I believe creating a whole new class for just that is a bit of an overkill.

My question thus is rather: how would I go about having one single function pointer (doesn't have to be of Print class) that can point to the write functions of different classes. I've read about pointer-to-function-members, but understood that they're bound to a specific class and cannot be generic.

My approach now is to have a one-liner plain-c function wrapper for each write function and have a regular c function pointer. Of course with additional context switches, so not so good. Is there a better way?

uint16_t (*writeFunPtr) (const uint8_t buf*, uint16_t len);
WifiClient tcpClient; // deliberately not a pointer! 
AsyncClient tcpAsyncClient; // deliberately not a pointer! 

uint16_t serWriWrap(const uint8_t buf*, uint16_t len) { return Serial.write(buf, len); }
uint16_t tcpWriWrap(const uint8_t buf*, uint16_t len) { return tcpClient.write(buf, len); }
uint16_t asyncWriWrap(const uint8_t buf*, uint16_t len) { return tcpAsyncClient.write(buf, len); }

void setup() { /*...*/ }
void loop() {
  writeFunPtr=&serWriWrap; writeFunPtr("SER", 4); // -->serial out, streamed
  writeFunPtr=&tcpWriWrap; writeFunPtr("TCP", 4); // -->tcp out, 1 packet
  writeFunPtr=&asyncWriWrap; writeFunPtr("ASYNC", 6); // -->async out, 1 packet
}

But other than that I think this can be closed if more people tested you code and @igrr puts it into master (preferably with the examples ;) ).

[igrr]

@nouser2013 see second and third recipe here: http://www.esp8266.com/viewtopic.php?p=39201#p39201

[nouser2013]

Oh boy, that's perfect. Thank you very much, I'll try it as soon as I'm home. me<=happy :)

[mph070770]

My apologies for opening this old thread but it seems to exactly reflect the issues i'm having with client.write. I've used the "me-no-dev" library and the code below, and that decreases the file transfer from 50 seconds to 20 seconds The file is only 150KB - why does it take an age to upload to the server? The same code on my PC takes no more than 2 seconds (including the server response). I know the ESP is not a PC - but from a networking perspective, where is the ESP bottleneck? Is there anything I can do to speed up the upload? My project is effectively dead before it's started if this is the best upload speed I can achieve :-(

#define MTU_SIZE 1460 
char buf[MTU_SIZE];

unsigned long startTime = millis();
      SPIFFS.begin();
      File spiFile = SPIFFS.open("/bigfile.bin", "r");
      if (!spiFile) USE_SERIAL.println("file open failed");
      else          USE_SERIAL.println("file opened");

      int siz = spiFile.size();
      while(siz > 0) {
        size_t len = std::min((int)(sizeof(buf)- 1), siz);
        spiFile.read((uint8_t *)buf, len);
        client.write((const uint8_t*)buf, len);
        siz -= len;
      }
      spiFile.close();
      unsigned long endTime = millis();
      USE_SERIAL.print("Duration:");
      USE_SERIAL.println(endTime-startTime);

[igrr]

You may try increasing buffer size to 2*MTU_SIZE, this may increase the speed somewhat with code we have in master branch (if you are using WiFiClient). The bottleneck is TCP window size. A desktop can easily handle 64kb window size, while ESP only has 2*MTU_SIZE window size. With small window size, latency limits the throughput you can get. In the edge case of a 1*MTU_SIZE window, you can only get MTU_SIZE/(2*latency) throughput.

[nouser2013]

@igrr isn't that hardcoded in the *.a from expressif?

his buf ist just the temp buffer to store the bin data in and does not concern LWIP window size(?). Perhaps he should declare buf whintin the loop in order to allocate it each loop run, thus giving LWIP effectively two buffers? then he can send non-blocking. alas, I still don't know what arguments to give client.write() to make it non-blocking...

Also, what OS is the remote server running?

[igrr]

I'm not suggesting to change MTU size (which, btw, it's not hardcoded in an .a file, we are using open-source lwip now, there is "build lwip from source" option in boards menu). I am suggesting to change the size of the chunk used to pass data to LwIP. With the latest code in master, WiFiClient can send up to 2*MTU_SIZE bytes at once. In fact, it may be better to do WiFiClient.write(spiFile), and WiFiClient will deal with splitting and sending.

[nouser2013]

In fact, it may be better to do WiFiClient.write(spiFile), and WiFiClient will deal with splitting and sending.

Perfect. Didn't know that this is possible now. Assuming it's thread-safe :)

[mph070770]

Thanks @igrr and @nouser2013 for your very quick replies!

@igrr - so my options appear limited? Is the MTU size a HW limitation? I had seen many posts online regarding the bandwidth of the ESP - such as this one - https://github.com/esp8266/Arduino/issues/1853#issuecomment-258540306 - so I assume these speeds are realised because the client and server have low latency (maybe on the same network) and the client gets very fast ACKs? I had already discussed this issue with @martinayotte, as I had tried the code you suggested in your second post (WiFiClient.write(spiFile) and it didn't seem to work - only 1 byte was transferred. See here: http://www.esp8266.com/viewtopic.php?p=57668#p57668. I'll double check I'm using the correct library.

@nouser2013 I orginally had by buf in the loop but the code was crashing - I assumed my stack was overflowing? If there's a performance improvement there, and WiFiClient.write(spiFile) doesn't work, I'll give it another go.

[andig]

@igrr is there any news on upgrading lwip to a more current version?

[igrr]

@andig when I have some news on this, I will reply to you on the relevant ticket :) For now updating LwIP for the 8266 non-OS SDK is in backlog.

Re throughput: window size is not a HW limitation, it's just the way LwIP is configured. You may try tweaking the windoow size but in this case you will probably have to reduce the number of TCP PCBs to avoid running out of RAM. Also you may need to change other LwIP options like the number of segments in flight.

[mph070770]

@igrr - "You may try tweaking the windoow size...." that's probably beyond my knowledge. You mentioned to @andig that updating to LwIP for the no-OS SDK is in backlog. Is there an inference that this isn't an issue for the RTOS SDK? What about the official Expressif SDK?

[igrr]

LwIP version update is almost entirely unrelated to the issue you are describing. I have mentioned non-OS SDK just to be specific. I don't remember if LwIP update for the ESP8266 FreeRTOS SDK is planned. Both SDK are official Espressif SDKs.

[mph070770]

ok, thanks. So just to confirm, my query above starting "my options appear limited" queried how other users had reported better throughput. Is that just because they had lower network latencies?

[martinayotte]

@mph070770 , I've done a test with the code I've already provided to you :

I've dump a 180K file over TCP Telnet, and it took less than 4 secs. I don't know why it takes more than 20 secs for you.

Make sure that you have a call to client.setNoDelay(1); right at the beginning of new connection.

[igrr]

I would suppose so. Also SPIFFS has some read latency, although I heard this was improved in recent versions. You may add some benchmarking to the code to see how much does SPIFFS contribute to overall latency. The speed you mentioned translates to approximately 400ms round-trip latency. Is this really the case? Can you try confirming this with wireshark?

[mph070770]

I'll do some investigations. The server is remote (it's a 3rd party server that I have no control of) and I don't (yet) know the performance of it so I'll check.

[martinayotte]

@mph070770 , you are confusing us : you were mentioning bad performance of SPIFFS file dump over TCP, so how this involve any remote server ?