search

s00500 / ESPUI

A simple web user interface library for ESP32 and ESP8266
https://valencia.lbsfilm.at/midterm-presentation/
Other
932 stars 174 forks source link

reaching the maximum of ressources? #232

Closed dewenni closed 1 year ago

dewenni commented 1 year ago

Hi,

first of all a big thanks for this amazing library! I have used it a project here: https://github.com/dewenni/ESP_Buderus_KM271 and it is a very nice addon to the project.

But now it seems that I have reached the limit of some ressources. I have used it to display a lot of parameters.

for example:

image

each row is based on 3 Label controls (description, value, unit). That is why i got a huge number of controls in this project. At the moment there are aprox 350 controls and it is nearly stable. But if I add some more controls >500, I run into problems.

It seems that, with >500 controls, there is a problem with the webserver and also with the WiFi Setup. At this point, the webUI does not start propper and also the WiFi of the ESP can not connect to the Router anymore.

Sometimes I get this error messages: [WiFiClient.cpp:422] write(): fail on fd 49, errno: 11, "No more processes"

Maybe the setup of the webUI is colliding with the WiFi Setup?

Does anybody had similar issues in the past or does anybody have some advice for me, how to handle such a huge number of controls?

I have already tried to increase: ESPUI.jsonUpdateDocumentSize = 2000; ESPUI.jsonInitialDocumentSize = 8000;

But it seems not to help.

I would appreciate help and support 🙏

MartinMueller2003 commented 1 year ago

You might want to take a look at the free heap size and the fragmentation in the heap. Your symptoms are the same as those when the memory is fragmented and no single block is available to send messages. I also have greater than 400 objects and had to do a bit of work to free up RAM to get things to work. Make sure you are not creating duplicates of your messages (use references whenever you can) and make sure any classes owning the UI objects free up strings after they have been sent to the UI.

dewenni commented 1 year ago

thanks for your answer!

I did some tests in a new and empty project, but I could not gain any significant insights.

Analysis

My test code was:

void setup() {

  Serial.begin(115200);
  while(!Serial) {} // Wait

  WiFi.mode(WIFI_STA);
  WiFi.begin("xxx", "xxx");
  WiFi.hostname("webUI-Test");
  while(WiFi.status() != WL_CONNECTED) {} // Wait

  uint16_t widgets, testtab[50];

  for (int tabs = 0; tabs <11; tabs++) {
    testtab[tabs] = ESPUI.addControl(ControlType::Tab, "", (String("Tab")+String(tabs)), ControlColor::None, 0);
    for (int i = 0; i < 100; i++) {
      ESPUI.addControl(Label, "", (String("label")+String(i)), None, testtab[tabs]);
      widgets++;
    }
  }

  //ESPUI.jsonUpdateDocumentSize = 2000;
  //ESPUI.jsonInitialDocumentSize = 8000;

  ESPUI.setVerbosity(Verbosity::Verbose);

  Serial.print("Widgets: ");
  Serial.println(widgets);

  ESPUI.begin("Test");
  Serial.println("Webserver started");

  Serial.print("HeapSize: ");
  Serial.println(ESP.getHeapSize());
  Serial.print("FreeHeap: ");
  Serial.println(ESP.getFreeHeap());
  Serial.print("MaxAllocHeap: ");
  Serial.println(ESP.getMaxAllocHeap());
  Serial.print("MinFreeHeap: ");
  Serial.println(ESP.getMinFreeHeap());

}

results:

Widgets: 1100 UI Initialized Webserver started HeapSize: 301908 FreeHeap: 101548 MaxAllocHeap: 53236 MinFreeHeap: 101304

=> works fine!

Widgets: 1200 UI Initialized Webserver started HeapSize: 300284 FreeHeap: 89828 MaxAllocHeap: 53236 MinFreeHeap: 89584

=> takes some browser refresh to get it load

Widgets: 1300 UI Initialized Webserver started HeapSize: 298660 FreeHeap: 78064 MaxAllocHeap: 53236 MinFreeHeap: 77820

=> crash

ESPUIclient::OnWsEvent:WS_EVT_CONNECT 1 Control::onWsEvent:No callback found for ID 1112 WS_EVT_DISCONNECT

abort() was called at PC 0x4013fa3b on core 1

#0  0x400836bd:0x3fff99d0 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402
  #1  0x4008b6e1:0x3fff99f0 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:128
  #2  0x40090d61:0x3fff9a10 in abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/abort.c:46
  #3  0x4013fa3b:0x3fff9a90 in __cxxabiv1::__terminate(void (*)()) at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
  #4  0x4013fa82:0x3fff9ab0 in std::terminate() at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
  #5  0x4013fe55:0x3fff9ad0 in __cxa_allocate_exception at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:300
  #6  0x4013f6e4:0x3fff9af0 in operator new(unsigned int) at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_op.cc:54
  #7  0x40145c76:0x3fff9b10 in AsyncWebServerRequest::beginResponse_P(int, String const&, unsigned char const*, unsigned int, std::function<String (String const&)>) at .pio/libdeps/esp32/ESP Async WebServer/src/WebRequest.cpp:764
  #8  0x400d5495:0x3fff9b50 in std::_Function_handler<void (AsyncWebServerRequest*), ESPUIClass::begin(char const*, char const*, char const*, unsigned short)::{lambda(AsyncWebServerRequest*)#4}>::_M_invoke(std::_Any_data const&, AsyncWebServerRequest*&&) at .pio/libdeps/esp32/ESPUI/src/ESPUI.cpp:1328
      (inlined by) _M_invoke at /Users/xxx/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:297
  #9  0x40149253:0x3fff9ba0 in std::function<void (AsyncWebServerRequest*)>::operator()(AsyncWebServerRequest*) const at /Users/xxx/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:687
  #10 0x4014928a:0x3fff9bd0 in AsyncCallbackWebHandler::handleRequest(AsyncWebServerRequest*) at .pio/libdeps/esp32/ESP Async WebServer/src/WebHandlerImpl.h:132
  #11 0x4014646d:0x3fff9c20 in AsyncWebServerRequest::_parseLine() at .pio/libdeps/esp32/ESP Async WebServer/src/WebRequest.cpp:581 (discriminator 1)
  #12 0x4014652d:0x3fff9c70 in AsyncWebServerRequest::_onData(void*, unsigned int) at .pio/libdeps/esp32/ESP Async WebServer/src/WebRequest.cpp:123
  #13 0x401467e5:0x3fff9cd0 in std::_Function_handler<void (void*, AsyncClient*, void*, unsigned int), AsyncWebServerRequest::AsyncWebServerRequest(AsyncWebServer*, AsyncClient*)::{lambda(void*, AsyncClient*, void*, unsigned int)#8}>::_M_invoke(std::_Any_data const&, void*&&, AsyncClient*&&, std::_Any_data const&, unsigned int&&) at .pio/libdeps/esp32/ESP Async WebServer/src/WebRequest.cpp:76
      (inlined by) _M_invoke at /Users/xxx/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:297
  #14 0x4014babf:0x3fff9cf0 in std::function<void (void*, AsyncClient*, void*, unsigned int)>::operator()(void*, AsyncClient*, void*, unsigned int) const at /Users/xxx/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:687
  #15 0x4014bb0a:0x3fff9d20 in AsyncClient::_recv(tcp_pcb*, pbuf*, signed char) at .pio/libdeps/esp32/AsyncTCP/src/AsyncTCP.cpp:915
  #16 0x4014bb4d:0x3fff9d40 in AsyncClient::_s_recv(void*, tcp_pcb*, pbuf*, signed char) at .pio/libdeps/esp32/AsyncTCP/src/AsyncTCP.cpp:1191
  #17 0x4014bc95:0x3fff9d60 in _handle_async_event(lwip_event_packet_t*) at .pio/libdeps/esp32/AsyncTCP/src/AsyncTCP.cpp:159
  #18 0x4014bd1e:0x3fff9d80 in _async_service_task(void*) at .pio/libdeps/esp32/AsyncTCP/src/AsyncTCP.cpp:194

In my real project the Limit was not at 1300 controls. It was already somewhere between 500-700. Of course, the more controls used, the smaller the heap-size. But I don't see the size becoming a problem. Because I didn't know what else to do, I started to rework the project.

re-design

I did some redesign of the webUI to get a lower number of controls. In the example above, I created some kind of table with a lot of single controls. Now a use only one control and use custom styles to get the same table style output.

Maybe it is also interesting for someone else.

I have defined some style in that way:

#define CUSTOM_CSS "<style>\
  .d30 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    background-color: unset;\
    width: 30\%;\
    text-align: left;\
  }\
  .d60 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    background-color: unset;\
    width: 60\%;\
    text-align: left;\
  }\
  .v30 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    width: 30\%;\
  }\
  .v40 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    width: 40\%;\
  }\
  .v70 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    width: 70\%;\
  }\
  .u10 {\
    box-sizing: border-box;\
    white-space: nowrap;\
    border-radius: 0.2em;\
    padding: 0.12em 0.4em 0.14em;\
    text-align: center;\
    color: #ffffff;\
    font-weight: 700;\
    line-height: 1.3;\
    margin-bottom: 5px;\
    display: inline-block;\
    white-space: nowrap;\
    vertical-align: baseline;\
    position: relative;\
    top: -0.15em;\
    background-color: #999999;\
    margin-bottom: 10px;\
    white-space: pre-wrap;\
    word-wrap: break-word;\
    width: 10\%;\
    background-color: unset;\
    text-align: left;\
  }\
 </style>"

to add this style definition I used a simple Label control that is hidden. 

// add additional css styles to a hidden label
ESPUI.setPanelStyle(ESPUI.label("CUSTOM CSS INJECTION", ControlColor::None, CUSTOM_CSS), "display: none");

With that little trick, I can use my own style definition with the normal Label controls.

In combination with this little "helper functions" I can create the same look and style of the shown example.

void initElements(){
  snprintf(elementBuffer, sizeof(elementBuffer), "");
}
void addElement(const char* label, const char* value){
  snprintf(elementString, sizeof(elementString), "<span class=\"d60\">%s</span><span class=\"v40\">%s</span>",  label, value);
  strcat(elementBuffer, elementString);
}
void addElementUnit(const char* label, const char* value, const char* unit){
  snprintf(elementString, sizeof(elementString), "<span class=\"d60\">%s</span><span class=\"v30\">%s</span><span class=\"u10\">%s</span>",  label, value, unit);
  strcat(elementBuffer, elementString);
}
void addElementWide(const char* label, const char* value){
  snprintf(elementString, sizeof(elementString), "<span class=\"d30\">%s</span><span class=\"v70\">%s</span>",  label, value);
  strcat(elementBuffer, elementString);
}
uint16_t addEmtyElement(uint16_t parent){
  uint16_t id = ESPUI.addControl(ControlType::Label, "", "--", ControlColor::None, parent);
  ESPUI.setElementStyle(id, "background-color: unset; width: 100%");
  return id;
}
void updateElements(uint16_t parent){
  ESPUI.updateLabel(parent, elementBuffer);
}

In the setup routine I only create an empty label.

// create table1
id.table1 = addEmtyElement(parentID);

and the update function looks like:

// update table1
    initElements();
    addElementUnit("description1",  "value1", "°C");
    addElementUnit("description2",  "value2", "°C");
    addElementUnit("description3",  "value3", "°C");
    ...
    updateElements(id.table1);

After that redesign I could decrease the amount of controls to a number of max. 300. The webUI reacts now significant faster and seems to be very reliable!

Maybe this is not the "smartest" way, bit it seems to work well in my case.

MartinMueller2003 commented 1 year ago

Actually, this is a memory allocation issue. The lines show that an alloc fails while trying to build a message to send to the browser and the system panics.

5 0x4013fe55:0x3fff9ad0 in __cxa_allocate_exception at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:300

6 0x4013f6e4:0x3fff9af0 in operator new(unsigned int) at /Users/brnomac003/.gitlab-runner/builds/qR2TxTby/0/idf/crosstool-NG/.build/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/new_op.cc:54

7 0x40145c76:0x3fff9b10 in AsyncWebServerRequest::beginResponse_P(int, String const&, unsigned char const*, unsigned int, std::function<String (String const&)>) at .pio/libdeps/esp32/ESP Async WebServer/src/WebRequest.cpp:764

8 0x400d5495:0x3fff9b50 in std::_Function_handler<void (AsyncWebServerRequest), ESPUIClass::begin(char const, char const, char const, unsigned short)::{lambda(AsyncWebServerRequest)#4}>::_M_invoke(std::_Any_data const&, AsyncWebServerRequest&&) at .pio/libdeps/esp32/ESPUI/src/ESPUI.cpp:1328

  (inlined by) _M_invoke at /Users/xxx/.platformio/packages/toolchain-xtensa-esp32@8.4.0+2021r2-patch5/xtensa-esp32-elf/include/c++/8.4.0/bits/std_function.h:297
dewenni commented 1 year ago

yes I saw that there is something with "alloc" but I could not see that it is related to critical less free heap-size

Widgets: 1100 HeapSize: 301908 FreeHeap: 101548 MaxAllocHeap: 53236 MinFreeHeap: 101304 => works fine!

Widgets: 1200 HeapSize: 300284 FreeHeap: 89828 MaxAllocHeap: 53236 MinFreeHeap: 89584 => takes some browser refresh to get it load

Widgets: 1300 HeapSize: 298660 FreeHeap: 78064 MaxAllocHeap: 53236 MinFreeHeap: 77820 => crash

MartinMueller2003 commented 1 year ago

Your heap report is made prior to most of the data being transferred to the browser. Once a browser connects, the resource usage goes up a bit and the memory starts to fragment. You could track that by adding a print after each response gets printed.