[idea] queue multiple things and print all with one at+cipsend

[atesin]

imagine an object like BatchPrinter or something, that accumulates pointers to strings, fstrings, numbers, etc, and then prints all at once in one at+cipsend command

i mean formalize something like what i tried to do in https://github.com/jandrassy/WiFiEspAT/issues/5#issuecomment-560035459 and https://github.com/jandrassy/WiFiEspAT/issues/5#issuecomment-560852532

imagine you make a compound http response with elements of many types taken from everywhere (pointers, no buffer), so you can queue, sum the lengths and print all in sequence at once

#define S const __FlashStringHelper *

WiFiClient myClient = httpClientWaitingResponse;

S welcome = F("<h1>Welcome ");
char user[] = "John Doe";
S notif = F(
  " <h1>"
  "<h2>you have "
);
int unread = getUnreadMessages();
S foot = F(" unread messages</h2>");

StackPrinter printerContents = myClient.newMassPrinter();
printerContents.add(welcome);
printeContentsr.add(user);
printeContentsr.add(notif);
printerContents.add(unread);
printerContents.add(foot);

StackPrinter printerHeaders = myClient.newMassPrinter();
printerHeaders.add(F(
  "HTTP/1.1 200 OK\r\n"
  "Content-Length: "
);
printerHeaders.add(printerContents.length());
printerHeaders.add("\r\n\r\n");

printerHeaders.printAll(myClient);
printerHeaders.destroy();
printerContents.printAll(myClient);
printerContents.destroy();

internally could work something like this (pseudocode java style, i am not so skilled in c[++]?)...

class BurstPrinter
{
  private int length = 0;
  private Object *jobs[] = new *Object[](); // some way to store different type pointers?, or with many arrays?

  public void add(Object obj) // overloaded method, or at least for char, char*, Fstr, byte, int, long, float, double ([un]signed)
  {
    length += obj.length();
    jobs.put(&obj);
  }

  public int length()
  {
    return this.length;
  }

  public void printAll(WiFiClient client)
  {
    client.print(F("AT+CIPSEND=");
    client.print(client.linkId);
    client.print(F(","));
    client.print(length);
    client.println();
    client.find('>');

    for (byte i = 0; i < jobs.size(); ++i) // <--- here the magic happens
      client.print(&jobs[i]);
  }

  public void destroy()
  {
    this.length = 0;
    this.jobs = null;
    this = null; // dont laugh ok?
  }
}

did you catch the idea ???

[JAndrassy]

you can't do it with with write(callback)?

[atesin]

i was looking the code (*) and i guess no, because write(callback) adds overheads and delays as it uses one at+cipsend for each print(), especially if composite output is made from several variables... and there is no way to know the lenght of data to send BEFORE send it (for Content-Length for example) (...... and uses buffers that will eventually fill?)

(*) according my limited skills

it could work like "output buffer" functions from php (ob_start(), ob_flush(), etc) but with an array of pointers https://www.php.net/manual/en/ref.outcontrol.php

... i see you are way more expert than me, sorry if you feel i waste your time with my dumb questions

[JAndrassy]

everything in the callback goes with on cipsendex. it doesn't require the length

[atesin]

everything in the callback goes with on cipsendex. it doesn't require the length

• 1 at+cipsendex for each print() right? (adds overheads and delays in callback functions with many nested print()'s )
  • overhead: https://github.com/jandrassy/WiFiEspAT/blob/3ba2e69385a844ee78d498bb8f6edec2fb2c6f5e/src/utility/EspAtDrv.cpp#L1182-L1193
  • delay (specifically line 1196): https://github.com/jandrassy/WiFiEspAT/blob/3ba2e69385a844ee78d498bb8f6edec2fb2c6f5e/src/utility/EspAtDrv.cpp#L1195-L1197
• you could need to know the data length before send the data itself, for example to send Content-Length header

what if you instead do something like

AT+CIPSEND=(link),(sum_of_str_lengths); // just 1 overhead, exact length (early available for previous uses)
find(">");
for str in strPtrArray: // iterate list of data to print
  client.print(&str);
// done

[JAndrassy]

just trust me. everything in one callback() goes with one cipsendex. there is AT+CIPSENDEX=, >, callback(), and \0 to finish. turn on debug and see.

for unknown http content-length chunked transfer encoding can be used. see the SDWebBrowser example

[atesin]

yes i know, but chunked encoding adds even more overheads and delays https://zoompf.com/blog/2012/05/too-chunky/

i would like to send compound data (a crafted webpage), of known length (for previous Content-Length header) in just 1 cipsend internal command... light and fast, no chunks, no overheads, no delays, no multiple commands, etc.

i think it could be possible to do it in an elegant way (... i did it but in the 'rough' way xD )

https://github.com/jandrassy/WiFiEspAT/issues/5#issuecomment-560852532 , see the last code block

  // send response

  byte link = client.id(); // this is the patch i am testing!

  esp.print(F("AT+CIPSEND="));
  esp.print(link);
  esp.print(',');
  esp.println(strlen_P((PGM_P) headers) + strlen_P((PGM_P) htdoc) + strlen(ipAddr));
  esp.find('>');

  esp.print(headers);
  esp.print(htdoc);
  esp.print(ipAddr);
  esp.find("OK\r\n");

  client.stop();
  echo(F("client served and disconnected: ") _ ipAddr);

[JAndrassy]

what delays with chunked encoding?

you can code it using the write(callback). just put

 esp.print(headers);
  esp.print(htdoc);
  esp.print(ipAddr);

in the callback

[atesin]

here 'esp' is the direct serial connection to esp8266 as seen in the full code block in https://github.com/jandrassy/WiFiEspAT/issues/5#issuecomment-560852532

i think each myCallback.print() in wifiClient.write(myCallback) ends each with an internal at+cipsendex command... so if i have

  myCallback.print(headers);
  myCallback.print(htdoc);
  myCallback.print(ipAddr);

i will end finally with 3 at+cipsendex, with 3 overheads and 3 20ms delays

additionally i have no way to previously know the data length to send the Content-Length header to avoid the browser waiting for more data and finally timeout (delays, awful user experience)

in addition, with chunked encoding i also have to code print hexadecimal sizes and add crlf terminators to each chunk, and add a dummy empty chunk at the end ("0\r\n")... even more overheads

[JAndrassy]

no, one calback is one cipsendex. the parameter of callback is the Serial to esp after AT+CIPSENDEX. see the SDWebBrowser example

[atesin]

so do you mean i can do something like.... (with 2 write()'s = 2 cipsendex + 2 20ms delays) ???

#define S const __FlashStringHelper *
S htdoc = F("your ip address is: ");

char ipAddr[16]; // ip address could be variable length
sprintf_P(ipAddr, P("%u.%u.%u.%u"), ip[0], ip[1], ip[2], ip[3]);

myClient.write([](Print& w)
{
  w.println(F("HTTP/1.1 200 OK"));
  w.println(F("Server: WiFiEspAT"));
  w.println(F("Connection: close"));
  w.print(F("Content-Length: "));
  w.println(strlen_P((PGM_P) htdoc) + strlen(ipAddr));
  w.println();
});

myClient.write([](Print& w)
{
  w.println(htdoc);
  w.println(ipAddr);
});

i think it can work, but still i found more efficient at+cipsend with precalculated exact lengths_sum (no 20ms delay)

i thought there is one at+cipsendex command for each w.print{ln}() command instead one for the whole callback fn.... the cipsendex is in executed in each inherited Print call inside callback or just once at the beginning of the callback?

side question: how can i guess the length of a decimal number representation (integer, floating point, any presicion, signed or not), in case of Content-Length: <length(myNumValue)> and later w.print(myNumValue)) ??? ... maybe 'Print' to some other 'buffer' just to measure the length... or maybe use the same ultoa(), etc than Print()'s

[atesin]

ok i got some crazy ideas

with java you could just create an Object[] array and later it will 'autodetect' data type, same with c++ (maybe, idk) with ctype... but ctype is not enabled by default in arduino c++ compiler to save resources (sram, flash and processing?), adittionally is unlikely many users will enable it just for this lib, so better find some alternative

we can create a class or struct with 2 arrays, one for pointers themselves and another to register data types, with overloaded methods to add() and print() each data

i think as every pointer points to a memory address it always takes 2 bytes (uint16_t?) whathever data type is (void?)... data types array could be just a char with b, i, l, f, d, c, p, respectively for byte, int, long, float, double, char, (cstr?) and pstr (with uppercase=unsigned?)

we could write many overloaded add() methods to appropiately increment this.length and fill both arrays, according entered data type... length() could be called at any time, same as printAll()

... so you can define the class with 10 or 20 elements in each aray or better pass it as parameter... then you can start to add() variables to print and it starts to increment this.length and populate arrays accordingly... so when you printAll() / printlnAll() it sends just one at+cipsend cmd with current accumulated this.length and then iterate over arrays with rawSerial.print()'s

... if everything go well actual printed variables should match with informed this.length and cipsend should end with 'recv ok' and ready again

[JAndrassy]

you described CStringBuilder from my StreamLib, only it copies everything into the buffer. the simplest is still codding the printing directly to output in callback. chunked encoding is fine if the resulting size is unknown. if you want to optimize to extreme, you can't use any library.

for JSON format ArduinoJson library collects pointers and then can print the final JSON to any Print (Serial, Client, ...). to get the size it does a dummy printing which only counts the chars.

EDIT: if you write a normal function for the callback, then you can run it first with a character counter Print implementation

[atesin]

you described CStringBuilder from my StreamLib, only it copies everything into the buffer

i dont mean copy to a buffer because the buffers are limited, you have the data duplicated twice what consumes ram and the copy process uses cpu time... instead using pointers to print directly from the source without any intermediate buffer

please... i mean using arrays of pointers instead of buffers, is faster, uses less ram, has no limit (just the original data), saves cpu time, etc

... pointers that points to data directly, no copy buffers before

for example my index page are compound of many F macro snippets of about 4kb total, with cstr/num variables in the middle.... my other pages are about 1k-2k... can't resist any buffer or copy process or chunked transfer

i dont see what is the problem of using direct pointers instead mid buffers... you gain speed, avoid limits, etc

pointers, no buffers, please

[JAndrassy]

then as I write in previous comment. write a function which sends this data to Print& out. then use the function to calculate the content length and as callback. a warning: cipsend and cipsendex are limited to 2 kB so one callback can only send less then 2kB.

[atesin]

an example of what i mean... the 'rough' way

#define S const __FlashStringHelper * // PGM_P POINTERS

#include "SoftwareSerial.h"
#include <WiFiEspAT.h>

SoftwareSerial esp(7, 8); // RX, TX
WiFiServer server(80);

void setup()
{
  esp.begin(19200); // enable software serial, connected to esp-01, (19200 works me flawless)
  WiFi.init(esp);
}

void loop()
{
  if ( !server.available() )
    return;

 WiFiClient client = server.available();
 if ( !validRequest(client) ) // just for the example
  return;

  char reqUrl[] = httpRequestUrl(client); // CHAR ARRRAY POINTERS
  char randomStr[] = someRandomStr();

  S header1 = F("HTTP/1.1 200 OK\r\n");
  S header2 = F("Server: WiFiEspAt\r\n");
  S header3 = F("Connection: close\r\n");
  S header4 = F("Content-Length: ");
  S header5 = F("\r\n\r\n");

  S content1 = F(
    "<http><head>"
    "<title>My Cool embedded http server!</title>"
    "</head><body>"
    "<h1>It works!</h1>"
    "<table border='0'>
    "<tr><td>Your requested url was : </td><td>"
  );
  S content2 = F(
    "</td></tr>"
    "<tr><td>Some random words of wisdom : </td>td>"
  );
  S content3 = F(
    "</td></tr>"
    "</table>"
    "</body></html>"
  );

  int contentLength =
    strlen_P((PGM_P) content1) +
    strlen(reqUrl)             +
    strlen_P((PGM_P) content2) +
    strlen(randomStr)          +
    strlen_P((PGM_P) content3) ;

  byte link = client.getLinkId(); // this is the patch... no need for this with a correct function/class implementation

  esp.print(F("AT+CIPSEND="));
  esp.print(link);
  esp.print(',');
  esp.println(
    strlen_P((PGM_P) header1) +
    strlen_P((PGM_P) header2) +
    strlen_P((PGM_P) header3) +
    strlen_P((PGM_P) header4) +
    getNumReprLengthSomeWay(contentLength) +
    strlen_P((PGM_P) header5) +
    contentLength
  );

  esp.find('>');

  esp.print(header1);
  esp.print(header2);
  esp.print(header3);
  esp.print(header4);
  esp.print(contentLength);
  esp.print(header5);
  esp.print(content1);
  esp.print(reqUrl);
  esp.print(content2);
  esp.print(randomStr);
  esp.print(content3);

  esp.find("OK\r\n");
  client.stop();
}  

[atesin]

then as I write in previous comment. write a function which sends this data to Print& out. then use the function to calculate the content length and as callback. a warning: cipsend and cipsendex are limited to 2 kB so one callback can only send less then 2kB.

i had some problems with this.... it seems static variables also turn it into constants, so the first header i sent was ok but the next were wrong... when i switched to traditional client.print()'s it worked fine (but slow)

[atesin]

i had some problems with this.... it seems static variables also turn it into constants, so the first header i sent was ok but the next were wrong... when i switched to traditional client.print()'s it worked fine (but slow)

my bad... i dont fully understands what means 'static' in c++ (in java there is also static vars and methods, sligtly different)

i experimented with wifiClient.write(callback) building dynamic http headers and html contents with no luck... i first defined as...

void httpSendHeaders(WiFiClient client, char statusCode0, int contentLength0)
{
  static char statusCode = statusCode0; // static to be accessible by lambda fn
  static int contentLength = contentLength0;

  client.flush();
  client.write([](Print& w) // anonymous lambda fn
  {
    w.print(F("HTTP/1.1 "));
    switch(statusCode)
    {
      case 'O':
        w.println(F("200 OK"));
// etc. ...

the problem was.... static variables remains 'static' in memory, i.e. it reserves its allocated memory the first (and only) time when declared, make them persistent, skipping this line in the next function calls...

so as i had static variable declaration and a value assignation in the same line, the assignation also run for once making actually a constant.... was hard to me to debug and understand, but when i separate the static declarations and the assignations in anothe line, the things started to flow

void httpSendHeaders(WiFiClient client, char statusCode0, int contentLength0)
{
  static char statusCode; // static to be accessible by lambda fn (memory allocated persistently, line called just once)
  static int contentLength;
  statusCode = statusCode0; // however it can be (re)reassigned after
  contentLength = contentLength0;

  client.flush();
  client.write([](Print& w) // anonymous lambda fn
  {
    w.print(F("HTTP/1.1 "));
    switch(statusCode)
    {
      case 'O':
        w.println(F("200 OK"));
        break;
      case 'N':
        w.println(F("404 Not Found")); // redirect to home?
        break;
      default:
        w.println(F("400 Bad Request"));
    }
    w.print(F(
      "Server: WiFiEspAT\r\n"
      "Connection: close\r\n"
      "Content-Length: "
    ));
    w.println(contentLength);
    w.println();
  });
}

it anyway has a 20ms delay each write(cb) invokation, but considering the costs and benefits i think is something i can live with =D

thank you @jandrassy