Wifi does not reconnect, memory leaks when reconnecting

Laxilef commented 8 months ago

Basic Infos

[x] This issue complies with the issue POLICY doc.
[x] I have read the documentation at readthedocs and the issue is not addressed there.
[ ] I have tested that the issue is present in current master branch (aka latest git).
[x] I have searched the issue tracker for a similar issue.
[x] If there is a stack dump, I have decoded it.
[x] I have filled out all fields below.

Platform

Hardware: ESP-12
Core Version: SDK:3.0.5(b29dcd3)/Core:3.1.2=30102000/lwIP:STABLE-2_1_3_RELEASE/glue:1.2-65-g06164fb/BearSSL:b024386
Development Env: Platformio
Operating System: Windows

Settings in IDE

Module: Wemos D1 mini
Flash Mode: default on platformio
Flash Size: 4MB
lwip Variant: v2 Lower Memory
Reset Method: default on platformio
Flash Frequency: 40Mhz
CPU Frequency: 80Mhz
Upload Using: OTA and SERIAL
Upload Speed: 115200

Problem Description

On some ESP8266 I am seeing a problem with reconnecting to Wifi. At a random moment in time, the ESP loses its connection to the Wifi and I try to reconnect, but the ESP does not connect. I also tried adding the following code before calling WiFi.begin():

        if (wifi_station_dhcpc_status() == DHCP_STARTED) {
          wifi_station_dhcpc_stop();
        }

        if (wifi_softap_dhcps_status() == DHCP_STARTED) {
          wifi_softap_dhcps_stop();
        }

        WiFi.setSleepMode(WIFI_NONE_SLEEP);
        WiFi.persistent(false);
        WiFi.softAPdisconnect();
        WiFi.disconnect(true, false);
        delay(200);
        WiFi.forceSleepWake();
        delay(100);
        WiFi.mode(WIFI_STA);
        delay(100);
        WiFi.setAutoConnect(true);
        WiFi.begin();

But this does not give any result. Eventually the memory runs out and the ESP reboots with an exception. Decoding the stacktrace does not give anything, because all the exceptions are related to the inability to allocate memory. The strange thing is that this does not happen with all ESPs.

I compiled with the flags -D DEBUG_ESP_CORE -D DEBUG_ESP_WIFI -D DEBUG_ESP_PORT=Serial and saw something strange:

state: 5 -> 2 (25a0)
rm 0
pm close 7
:er -13 0x00000000
wifi evt: 1
STA disconnect: 37
forcibly stopping the station connection manager
state: 2 -> 0 (0)
wifi evt: 1
STA disconnect: 202

After this, the ESP refuses to connect to Wifi. But after a restart (cold start), the connection works correctly.

MCVE Sketch

https://raw.githubusercontent.com/tzapu/WiFiManager/master/examples/NonBlocking/AutoConnectNonBlocking/AutoConnectNonBlocking.ino

Debug Messages

[01:40:43][MQTT][TRACE] Successfully publish 431 of 431 bytes to topic: opentherm/state
state: 5 -> 2 (25a0)
rm 0
pm close 7
:er -13 0x00000000
wifi evt: 1
STA disconnect: 37
forcibly stopping the station connection manager
state: 2 -> 0 (0)
wifi evt: 1
STA disconnect: 202
[01:40:43][MQTT][INFO] Disabled
[01:40:43][WIFI][INFO] Disconnected
[01:40:43][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8В
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:40:43][WIFI][TRACE] Successfully set mode '1'
[01:40:43][WIFI][TRACE] Hostname 'opentherm' is already set
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:40:48][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:40:48][WIFI][TRACE] Successfully set mode '1'
[01:40:48][WIFI][TRACE] Hostname 'opentherm' is already set
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:40:53][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:40:54][WIFI][TRACE] Successfully set mode '1'
[01:40:54][WIFI][TRACE] Hostname 'opentherm' is already set
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:40:57][MAIN][VERB] Free heap size: 15904 of 81920 bytes (min: 14144, diff: 0), max free block: 15808 (min: 7128, diff: 0, frag: 0%)
[01:40:59][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:40:59][WIFI][TRACE] Successfully set mode '1'
[01:40:59][WIFI][TRACE] Hostname 'opentherm' is already set
[01:40:59][MAIN][VERB] Free heap size: 14048 of 81920 bytes (min: 14048, diff: 96), max free block: 13952 (min: 7128, diff: 0, frag: 0%)
[01:41:00][MAIN][VERB] Free heap size: 13816 of 81920 bytes (min: 13816, diff: 232), max free block: 13720 (min: 7128, diff: 0, frag: 0%)
[01:41:00][MAIN][VERB] Free heap size: 13584 of 81920 bytes (min: 13584, diff: 232), max free block: 13488 (min: 7128, diff: 0, frag: 0%)
[01:41:00][MAIN][VERB] Free heap size: 13352 of 81920 bytes (min: 13352, diff: 232), max free block: 13256 (min: 7128, diff: 0, frag: 0%)
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:41:04][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:41:04][WIFI][TRACE] Successfully set mode '1'
[01:41:04][WIFI][TRACE] Hostname 'opentherm' is already set
[01:41:04][MAIN][VERB] Free heap size: 13120 of 81920 bytes (min: 13120, diff: 232), max free block: 13024 (min: 7128, diff: 0, frag: 0%)
[01:41:04][MAIN][VERB] Free heap size: 12888 of 81920 bytes (min: 12888, diff: 232), max free block: 12792 (min: 7128, diff: 0, frag: 0%)
[01:41:04][MAIN][VERB] Free heap size: 12656 of 81920 bytes (min: 12656, diff: 232), max free block: 12560 (min: 7128, diff: 0, frag: 0%)
[01:41:04][MAIN][VERB] Free heap size: 12424 of 81920 bytes (min: 12424, diff: 232), max free block: 12328 (min: 7128, diff: 0, frag: 0%)
[01:41:04][MAIN][VERB] Free heap size: 11960 of 81920 bytes (min: 11960, diff: 464), max free block: 11864 (min: 7128, diff: 0, frag: 0%)
[01:41:05][MAIN][VERB] Free heap size: 11728 of 81920 bytes (min: 11728, diff: 232), max free block: 11632 (min: 7128, diff: 0, frag: 0%)
[01:41:05][MAIN][VERB] Free heap size: 11496 of 81920 bytes (min: 11496, diff: 232), max free block: 11400 (min: 7128, diff: 0, frag: 0%)
[01:41:05][MAIN][VERB] Free heap size: 11264 of 81920 bytes (min: 11264, diff: 232), max free block: 11168 (min: 7128, diff: 0, frag: 0%)
[01:41:05][MAIN][VERB] Free heap size: 11032 of 81920 bytes (min: 11032, diff: 232), max free block: 10936 (min: 7128, diff: 0, frag: 0%)
[01:41:05][MAIN][VERB] Free heap size: 10800 of 81920 bytes (min: 10800, diff: 232), max free block: 10704 (min: 7128, diff: 0, frag: 0%)
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:41:09][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:41:09][WIFI][TRACE] Successfully set mode '1'
[01:41:09][WIFI][TRACE] Hostname 'opentherm' is already set
[01:41:09][MAIN][VERB] Free heap size: 10568 of 81920 bytes (min: 10568, diff: 232), max free block: 10472 (min: 7128, diff: 0, frag: 0%)
[01:41:09][MAIN][VERB] Free heap size: 10336 of 81920 bytes (min: 10336, diff: 232), max free block: 10240 (min: 7128, diff: 0, frag: 0%)
[01:41:09][MAIN][VERB] Free heap size: 10104 of 81920 bytes (min: 10104, diff: 232), max free block: 10008 (min: 7128, diff: 0, frag: 0%)
[01:41:10][MAIN][VERB] Free heap size: 9872 of 81920 bytes (min: 9872, diff: 232), max free block: 9776 (min: 7128, diff: 0, frag: 0%)
[01:41:10][MAIN][VERB] Free heap size: 9640 of 81920 bytes (min: 9640, diff: 232), max free block: 9544 (min: 7128, diff: 0, frag: 0%)
[01:41:10][MAIN][VERB] Free heap size: 9176 of 81920 bytes (min: 9176, diff: 464), max free block: 9080 (min: 7128, diff: 0, frag: 1%)
[01:41:10][MAIN][VERB] Free heap size: 8944 of 81920 bytes (min: 8944, diff: 232), max free block: 8848 (min: 7128, diff: 0, frag: 1%)
[01:41:10][MAIN][VERB] Free heap size: 8712 of 81920 bytes (min: 8712, diff: 232), max free block: 8616 (min: 7128, diff: 0, frag: 1%)
[01:41:10][MAIN][VERB] Free heap size: 8480 of 81920 bytes (min: 8480, diff: 232), max free block: 8384 (min: 7128, diff: 0, frag: 1%)
[01:41:10][MAIN][VERB] Free heap size: 8248 of 81920 bytes (min: 8248, diff: 232), max free block: 8152 (min: 7128, diff: 0, frag: 1%)
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:41:14][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:41:14][WIFI][TRACE] Successfully set mode '1'
[01:41:14][WIFI][TRACE] Hostname 'opentherm' is already set
[01:41:14][MAIN][VERB] Free heap size: 8016 of 81920 bytes (min: 8016, diff: 232), max free block: 7920 (min: 7128, diff: 0, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 7784 of 81920 bytes (min: 7784, diff: 232), max free block: 7688 (min: 7128, diff: 0, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 7552 of 81920 bytes (min: 7552, diff: 232), max free block: 7456 (min: 7128, diff: 0, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 7320 of 81920 bytes (min: 7320, diff: 232), max free block: 7224 (min: 7128, diff: 0, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 7088 of 81920 bytes (min: 7088, diff: 232), max free block: 6992 (min: 6992, diff: 136, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 6624 of 81920 bytes (min: 6624, diff: 464), max free block: 6528 (min: 6528, diff: 464, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 6392 of 81920 bytes (min: 6392, diff: 232), max free block: 6296 (min: 6296, diff: 232, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 5928 of 81920 bytes (min: 5928, diff: 464), max free block: 5832 (min: 5832, diff: 464, frag: 1%)
[01:41:15][MAIN][VERB] Free heap size: 5696 of 81920 bytes (min: 5696, diff: 232), max free block: 5600 (min: 5600, diff: 232, frag: 1%)
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:41:19][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:41:20][WIFI][TRACE] Successfully set mode '1'
[01:41:20][WIFI][TRACE] Hostname 'opentherm' is already set
[01:41:20][MAIN][VERB] Free heap size: 5464 of 81920 bytes (min: 5464, diff: 232), max free block: 5368 (min: 5368, diff: 232, frag: 1%)
[01:41:20][MAIN][VERB] Free heap size: 5232 of 81920 bytes (min: 5232, diff: 232), max free block: 5136 (min: 5136, diff: 232, frag: 1%)
[01:41:20][MAIN][VERB] Free heap size: 5000 of 81920 bytes (min: 5000, diff: 232), max free block: 4904 (min: 4904, diff: 232, frag: 1%)
[01:41:20][MAIN][VERB] Free heap size: 4768 of 81920 bytes (min: 4768, diff: 232), max free block: 4672 (min: 4672, diff: 232, frag: 2%)
[01:41:20][MAIN][VERB] Free heap size: 4304 of 81920 bytes (min: 4304, diff: 464), max free block: 4208 (min: 4208, diff: 464, frag: 2%)
[01:41:20][MAIN][VERB] Free heap size: 4072 of 81920 bytes (min: 4072, diff: 232), max free block: 3976 (min: 3976, diff: 232, frag: 2%)
[01:41:20][MAIN][VERB] Free heap size: 3840 of 81920 bytes (min: 3840, diff: 232), max free block: 3744 (min: 3744, diff: 232, frag: 2%)
[01:41:20][MAIN][VERB] Free heap size: 3608 of 81920 bytes (min: 3608, diff: 232), max free block: 3512 (min: 3512, diff: 232, frag: 2%)
[01:41:21][MAIN][VERB] Free heap size: 3376 of 81920 bytes (min: 3376, diff: 232), max free block: 3280 (min: 3280, diff: 232, frag: 2%)
[01:41:21][MAIN][VERB] Free heap size: 3144 of 81920 bytes (min: 3144, diff: 232), max free block: 3048 (min: 3048, diff: 232, frag: 3%)
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
wifi evt: 1
STA disconnect: 2
[01:41:25][WIFI][INFO] Reconnecting...
[APdisconnect] set_config failed!
del if0
usl
mode : null
wifi evt: 8
mode : sta(fc:f5:c4:96:39:dc)
add if0
wifi evt: 8
[01:41:25][WIFI][TRACE] Successfully set mode '1'
[01:41:25][WIFI][TRACE] Hostname 'opentherm' is already set
[01:41:25][MAIN][VERB] Free heap size: 2912 of 81920 bytes (min: 2912, diff: 232), max free block: 2816 (min: 2816, diff: 232, frag: 3%)
[01:41:25][MAIN][VERB] Free heap size: 2680 of 81920 bytes (min: 2680, diff: 232), max free block: 2584 (min: 2584, diff: 232, frag: 3%)
[01:41:25][MAIN][VERB] Free heap size: 2448 of 81920 bytes (min: 2448, diff: 232), max free block: 2352 (min: 2352, diff: 232, frag: 3%)
[01:41:25][MAIN][VERB] Free heap size: 2216 of 81920 bytes (min: 2216, diff: 232), max free block: 2120 (min: 2120, diff: 232, frag: 4%)
[01:41:25][MAIN][VERB] Free heap size: 1984 of 81920 bytes (min: 1984, diff: 232), max free block: 1888 (min: 1888, diff: 232, frag: 4%)
[01:41:25][MAIN][VERB] Free heap size: 1520 of 81920 bytes (min: 1520, diff: 464), max free block: 1424 (min: 1424, diff: 464, frag: 6%)
[01:41:26][MAIN][VERB] Free heap size: 1288 of 81920 bytes (min: 1288, diff: 232), max free block: 1192 (min: 1192, diff: 232, frag: 7%)

mcspr commented 8 months ago

Code above does not look like the linked WiFiManager .ino file though. You sure it is Core and / or SDK leaking something and not your app / WiFiManager / some other lib?

STA disconnect: 202

https://github.com/esp8266/Arduino/blob/d5eb265f78bff9deb7063d10030a02d021c8c66c/tools/sdk/include/user_interface.h#L523

Name would suggest it could not authenticate with the given passphrase and / or auth settings

WiFi.begin();

begin() without parameters usually means SSID & passphrase were already set, either in persistent mode when those are read from flash or without one where those are managed in RAM. Invalid credentials are the most likely explanation for the error above.

Laxilef commented 8 months ago

Code above does not look like the linked WiFiManager .ino file though. You sure it is Core and / or SDK leaking something and not your app / WiFiManager / some other lib?

I am using this code in my project OTGateway. I didn’t show the entire code of the project, because it wouldn’t be interesting to you, but part of the code for working with Wifi can be viewed here. I tried removing other code and working only with Wifi, the behavior is the same. At least 2 users reported me about this behavior and sent me logs. I spent a long time studying other used libraries and did not find any leaks there. It makes me think that the problem is in SDK due to the fact that memory leaks begin strictly after calling the reconnection code to Wifi.

It's really strange that I've very rarely been able to reproduce this, meaning it's not permanent. I've been trying to find the reason for more than 2 weeks before creating this issue.

begin() without parameters usually means SSID & passphrase were already set, either in persistent mode when those are read from flash or without one where those are managed in RAM. Invalid credentials are the most likely explanation for the error above.

Hmm. SSID and password are stored in flash memory. However, after a reboot or cold start, the ESP successfully connects to the router. For example, calling WiFi.reconnect() gave the same result: the ESP did not connect and the memory decreased with each call to this method.

What advanced tests can I run for you to investigate this issue? Could it be hardware problems? Thanks.

mcspr commented 8 months ago

Hmm. SSID and password are stored in flash memory. However, after a reboot or cold start, the ESP successfully connects to the router. For example, calling WiFi.reconnect() gave the same result: the ESP did not connect and the memory decreased with each call to this method.

Depends on two SDK settings, WiFi has slightly different names for them (see ESP8266WiFiSTA.cpp for the underlying stuff)

WiFi.setAutoConnect(true / false), WiFi.getAutoConnect for current setting - try to connect on boot. assuming this works ok, and already enabled
WiFi.setAutoReconnect, WiFi.getAutoReconnect - it would keep re-trying when connection attempt fails or after disconnecting from an already connected AP. No user interaction required, it is supposed to work without any extra calls like begin(), reconnect(), disconnect(), etc.

The minimal setup would look something like this, without WiFiManager for right now. Loop then never calls anything related to connection management

void setup() {
    // otherwise, Core turns WiFi off. this reverts to 2.x.x and SDK default STATION opmode
    enableWiFiAtBootTime();

    // so WiFi.begin(..., ...) puts stuff into flash
    WiFi.persistent(true);

    // note that persistent() does not affect these two, both values always persist between reboots
    if (!WiFi.getAutoConnect()) {
        WiFi.setAutoConnect(true);
    }

    if (!WiFi.getAutoReconnect()) {
        WiFi.setAutoReconnect(true);
    }

    // assume connection already happening. call begin to reset credentials
    if (WiFi.waitForConnectResult(10000) != WL_CONNECTED) {
        WiFi.begin(MY_SSID, MY_PASS);
    }
}

I don't see anything suspicious with a simple ping app. Also no difference between SDK versions. 2.2.x and then upgraded to 3.0.5, both seem to work just fine.

Heap slowly goes down from 50968 to 47056, but never lower. Disconnection is either by simply covering the antenna, or iw dev ... station del MAC kick from the router cli

Laxilef commented 8 months ago

Hi, @mcspr! Happy New Year and thanks for your reply. I also tried this option, but previously abandoned it because there were complaints that when disconnected from Wifi, the ESP did not reconnect. But I did not call begin when there was no connection, since it is assumed that the connection data is taken from flash.

I am now confused about the situation I described in the first msg. For tests, I’ll try to make the code without all the extra stuff and come back with the results. If you have any ideas why this might be happening, please let me know. Thanks!

esp8266 / Arduino