possible memory leak with rapid wifi queries

distrakt commented 4 months ago

Board

ESP32, ESP32-S2

Device Description

I'm using "mini" style boards of ESP32 and ESP32-S2

Hardware Configuration

none

Version

v3.0.0

IDE Name

ARDUINO

Operating System

macos

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

With apologies -- this is a heads-up and I have not narrowed it down. I am using library "OmEspHelpers" which implements a web server.

On versions 2.0.17 and earlier, memory is solid, and any number of web requests continue working, free memory varies but does not trend down.

On versions 3.0.0 and 3.0.2, web requests appear to drain several hundred bytes each request. [EDIT] After waiting a little while, memory comes back. This is still a hazard for rapid-fire requests such with rapid REST polling.

[EDIT] I just attached a basic web server example, which serves up "freeBytes: xxxxx". When I hit refresh on browser, freebytes trends downward consistently.

Sketch


#include <WebServer.h>
#include <WiFi.h>

/*Put your SSID & Password*/
const char *ssid = "omino warp";  // Enter SSID here
const char *password = "0123456789"; // Enter Password here

WebServer server(80);

void setup()
{
  Serial.begin(115200);
  delay(100);

  Serial.println("Connecting to ");
  Serial.println(ssid);

  // connect to your local wi-fi network
  WiFi.begin(ssid, password);

  // check wi-fi is connected to wi-fi network
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected..!");
  Serial.print("Got IP: ");
  Serial.println(WiFi.localIP());

  server.on("/", handle_OnConnect);
  server.onNotFound(handle_OnConnect);

  server.begin();
  Serial.println("HTTP server started");
}
void loop()
{
  server.handleClient();
}

void handle_OnConnect()
{
  server.send(200, "text/html", SendHTML());
}

String SendHTML()
{

  String ptr = "<!DOCTYPE html> <html>\n";
  ptr += "<head><meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, user-scalable=no\">\n";
  ptr += "<title>esp32</title>\n";
  ptr += "</head>\n";
  ptr += "<body>\n";
  ptr += "<h1>ESP32 Web Server</h1>\n";
  ptr += "<h3>Using Station(STA) Mode</h3>\n";

  char s[128];
  sprintf(s, "<pre><hr/>freeBytes: %d\n<hr/></pre>\n",  esp_get_free_heap_size());
  ptr += s;

  ptr += "</body>\n";
  ptr += "</html>\n";
  return ptr;
}

Debug Message

none
reading memory with system_get_free_heap_size()
eventually hangs

Other Steps to Reproduce

none

I have checked existing issues, online documentation and the Troubleshooting Guide

[X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

TD-er commented 4 months ago

Please also check if the used memory is later freed after +/- 2 minutes when you stop making the calls.

distrakt commented 4 months ago

TD-er with skepticism I checked... and you're right! Sure enough after a little while memory was freed. So that's both nice, and not-nice. For a REST-like service, calls still come in potentially quite fast (in my case, handling html slider movements), so a 2 minute memory cleanup turnaround is a step backwards from 2.x.

If this (garbage collector?) is configurable, I feel perhaps it ought to default to a short interval like before.

What is the name of this system/feature/api ... ?

TD-er commented 4 months ago

lbernstone commented 4 months ago

I don't really have the ability to verify, but this kind of thing can often happen if you get a bunch of sockets in TIME_WAIT or CLOSE_WAIT status because sessions aren't closing cleanly. Make sure you put appropriate headers on requests that should be closed.

TD-er commented 4 months ago

I wonder how much you can do on the ESP side and how much is not closed properly by tools like the browser.

distrakt commented 4 months ago

This feels plausible... it feels like a 60-second timeout thing going on.

In my case, the ESP is a server, handling http requests. It calls client.close() after replying.

In any case, though, it is a regression from 2.x, which does not show this memory-loss-and-later-recover. if it’s a new “improvement” with some compatibility issues it should be switchable perhaps?

VojtechBartoska commented 4 months ago

@me-no-dev please help with triage of this issue, thanks

me-no-dev commented 4 months ago

@VojtechBartoska I can't. Missing is a minimal reproducible example and instructions. It's all theory up to this point.

TD-er commented 4 months ago

Just serving some minimal HTML page and hitting it with a browser that does frequent refresh will cause this behavior. Then periodically output the amount of free memory and a timestamp will show a decrease per pageview. After stopping those calls you will gradually see the amount of free memory increase again, for each call that was 120 sec ago.

N.B. On nodes with PSRAM, make sure to also keep track of free PSRAM memory.

distrakt commented 4 months ago

As per TD-er -- I attached a basic web server example i found, on the page it prints the free bytes, and indeed shows the issue.

me-no-dev commented 4 months ago

Issue confirmed. Each request takes between 200 and 240 bytes. Those bytes are returned a minute or so later, which means that they are the connection in TIME_WAIT. It's interesting to me why the number of such sockets is not limited internally, as I was able to run the ESP almost out of memory by in effect DDoSing it with requests. The issue is somewhere below Arduino level though. I wonder if there is something we can do, like reuse client or something...

me-no-dev commented 4 months ago

issue is not present on 2.0.17. Such connection count is limited there to 16.

me-no-dev commented 4 months ago

Patch added in the lib-builder. Will be part of next release

Jason2866 commented 4 months ago

Tested the fix. Does work :-)

espressif / arduino-esp32