search

letscontrolit / ESPEasy

Easy MultiSensor device based on ESP8266/ESP32
http://www.espeasy.com
Other
3.28k stars 2.22k forks source link

Home Assistant (openHAB) MQTT controller is exhausting RAM during reconnect #2684

Closed ghtester closed 3 years ago

ghtester commented 5 years ago

I am testing a custom build (by Vagrant) from official sources 20190926 with a plugin set customized Custom.h.txt The firmware details: Build:⋄ | 20104 - Mega System Libraries:⋄ | ESP82xx Core 2.6.0-dev, NONOS SDK 2.2.2-dev(38a443e), LWIP: 2.1.2 PUYA support Git Build:⋄ | My Build: Sep 26 201908:20:00 Plugins:⋄ | 34 [Normal] Build Md5: | e6d1ef3eb972c31b6659c853501e48 Md5 check: | passed. Build Time:⋄ | Sep 26 2019 08:09:00 Binary Filename:⋄ | ESP_Easy_20190926_vagrant_custom_ESP8266_4M1M.bin

The ESP8266 node with weak WiFi reception (about -90 or even worse) is reconnecting to AP quite often. Without MQTT controller active it's not an issue but as soon as the MQTT controller is enabled and a data from sensor are transferred to MQTT broker, soon or later the RAM is exhausted (regardless on controller settings), node stops sending data and reboots in the end (usually with Exception 29). The current controller settings is here (started with defaults, tried to change everything without positive effect): MQTT I am sending data from BMP280 every 15 seconds. Also tried 2 different MQTT brokers (openHAB 2.5 embedded - unusable and external - CentOS 7 default) with the same issue.

TD-er commented 5 years ago

@giig1967g Nope, I have also reports of users for whom I made some builds and they report the same. Those were not built in Vagrant. I am now testing here on a few nodes by simulating bad WiFi connections. But the thing is those simulations only are about proper disconnects and I guess the real problem is when a disconnect is not 'clean'. So maybe there are outstanding network requests which are not cleared.

ghtester commented 5 years ago

@giig1967g Please note I am also filling the MQTT queue with BMP280 sensor's data every 15s and this is maybe the reason why the issue happen more frequently. The bad WiFi signal (RSSI about -90 or worse) is a must to see the MQTT related troubles.

ghtester commented 5 years ago

So the core 2.5.2 is no way for me, just tested it... The MQTT issue described above still persists (only the reboot reasons are various but it's for sure all about exhausting a memory), in addition the old issue with IRRX reliability is back. The plus is that reconnection to AP after warm reboots works fine..

TD-er commented 5 years ago

Could you test the changes I made here: https://github.com/letscontrolit/ESPEasy/pull/2728 ?

ghtester commented 5 years ago

I am afraid not as I don't know how to update the source tree (on Vagrant VM) properly when there are more changes.

TD-er commented 5 years ago

Within the vagrant environment, you can run tools/build_ESPeasy.sh -p 2728 from within the ESPEasy directory. This will create a zip file, which you then can copy to the mounted shared dir.

ghtester commented 5 years ago

OK, thanks for the details, I'll give it a try but should I first revert the changes I made earlier in files platformio.ini, Controller.ino and _P037_MQTTImport.ino?

TD-er commented 5 years ago

I started a build on my NUC server, so it will be ready in 45 minutes.

TD-er commented 5 years ago

Hmm, I am running the test version and it disconnects quite often, so don't bother to test it right now.

ghtester commented 5 years ago

OK, Gijs, thanks a lot for your effort.

TD-er commented 4 years ago

Can you test the testbuild linked here: https://github.com/letscontrolit/ESPEasy/pull/2728#issuecomment-553169609 ?

ghtester commented 4 years ago

Thanks a lot for the test build. I have installed the ESP_Easy_mega-20191108-36-PR_2728_test_core_260_sdk222_alpha_ESP8266_4M1M.bin. Although the test conditions are NOT the same as this build does not contain IRRX plugin so there's more free RAM available, it looks much better than earlier version. So far the node survived 88 Connect Failures without reboot, the FreeMem is decreasing and increasing (at boot 15680 , minimum 4944, currently about 11000). As soon as the patch is released, I'll test with custom build containing the plugins I need including IRRX (hopefully the Vagrant build does not mess the WiFi much). I am afraid about low FreeMem when the IRRX is used but we'll see...

TD-er commented 4 years ago

Do you have an indication on how many reconnects there were before until the unit rebooted?

ghtester commented 4 years ago

As mentioned earlier, in past usually about 20 - 40 reconnects was enough to reboot the node due to exhausted memory.
The current build survived 108 reconnects and just rebooted (FreeMem went below 2896). After reboot it can't reconnect to AP despite I removed the shield so the WiFi signal is good (reason: AP not found).

Edit: node reconnected to AP after about 90 attempts (there's backup AP configured but inaccessible in this location) but ESP node is still visible as AP as well (#2721).

TD-er commented 4 years ago

Do you have "reset wifi" flag checked in ESPEasy? 90 reconnect attempts is a lot.

ghtester commented 4 years ago

Yes but it looks it does not work as expected... See the part of the log:

... 2969689 : Info : EVENT: WiFi#Disconnected 2969911 : Info : WIFI : Disconnected! Reason: '(201) No AP found' Connected for 2856 ms 2969913 : Info : WIFI : Set WiFi to OFF 2969914 : Info : WIFI : Cannot set mode!!!!! 2970917 : Info : EVENT: WiFi#APmodeDisabled sleep disable 2971333 : Info : WIFI : Connecting MyTestAP attempt #93 2972412 : Info : BH1750 Address: 0x23 Mode: 0x1 : Light intensity: 16.67 2972414 : Info : EVENT: Light#Lux=16.67 2973434 : Info : BMP280 : Address: 0x76 2973436 : Info : BMP280 : Temperature: 26.03 2973438 : Info : BMP280 : Barometric Pressure: 982.02 2973440 : Info : EVENT: BMX280#Temperature=26.03 2973663 : Info : EVENT: BMX280#Humidity=0.00 2973863 : Info : EVENT: BMX280#Pressure=982.02 scandone state: 0 -> 2 (b0) state: 2 -> 0 (2) 2976476 : Info : EVENT: WiFi#Disconnected 2976697 : Info : WIFI : Disconnected! Reason: '(2) Auth expire' Connected for 4816 ms 2976699 : Info : WIFI : Set WiFi to OFF 2976701 : Info : WIFI : Cannot set mode!!!!! 2977704 : Info : EVENT: WiFi#APmodeDisabled sleep disable 2978120 : Info : WIFI : Connecting MyBackupAP attempt #94 2978135 : Info : WD : Uptime 50 ConnectFailures 8 FreeMem 2808 WiFiStatus 6 scandone 2981081 : Info : EVENT: WiFi#Disconnected 2981281 : Info : WIFI : Disconnected! Reason: '(201) No AP found' Connected for 2856 ms 2981283 : Info : WIFI : Set WiFi to OFF 2981284 : Info : WIFI : Cannot set mode!!!!! 2982287 : Info : EVENT: WiFi#APmodeDisabled sleep disable 2982701 : Info : WIFI : Connecting MyBackupAP attempt #95 scandone 2985663 : Info : EVENT: WiFi#Disconnected 2985866 : Info : WIFI : Disconnected! Reason: '(201) No AP found' Connected for 2856 ms 2985868 : Info : WIFI : Set WiFi to OFF 2985870 : Info : WIFI : Cannot set mode!!!!! 2986873 : Info : EVENT: WiFi#APmodeDisabled sleep disable 2987291 : Info : WIFI : Connecting MyTestAP attempt #96 ...

Now it is connected again, node rebooted due to low memory, this time yet after 12 Connect Failures (uptime 60 min). ESP node is NOT visible as AP anymore. I'll try to turn off "MQTT Retain Msg:" in Advanced settings to see if there's any change..

TD-er commented 4 years ago

Hmm that's strange, why would the ESP not be able to set the wifi mode? What core version are you using?

ghtester commented 4 years ago
ESP_Easy_mega-20191108-36-PR_2728_test_core_260_sdk222_alpha_ESP8266_4M1M.bin Build:⋄ 20104 - Mega
System Libraries:⋄ ESP82xx Core 2_6_0, NONOS SDK 2.2.2-dev(bb83b9b), LWIP: 2.1.2 PUYA support
Git Build:⋄  
Plugins:⋄ 79 [Normal] [Testing]
Build Md5: 25a33289a75d7da07dff343f1daf43d3
Md5 check: passed.
Build Time:⋄ Nov 13 2019 00:28:26
Binary Filename:⋄ ESP_Easy_mega-20191108-36-PR_2728_test_core_260_sdk222_alpha_ESP
ghtester commented 4 years ago

BTW. now the node rebooted due to different reason (while the FreeMem was high enough): ...... 3244809 : Error : WIFI : Error while starting AP Mode with SSID: ESP_05 IP: 192.168.4.1 3244919 : Info : Dummy: value 1: 1.00 3244920 : Info : EVENT: Time#1=1.00 3245119 : Info : EVENT: Time#2=0.00 3245317 : Info : EVENT: Time#3=0.00 3245517 : Info : EVENT: Time#4=0.00 3246287 : Info : WD : Uptime 54 ConnectFailures 2 FreeMem 9880 WiFiStatus 0 3258863 : Info : BMP280 : Address: 0x76 3258864 : Info : BMP280 : Temperature: 25.93 3258864 : Info : BMP280 : Barometric Pressure: 982.20 3258865 : Info : EVENT: BMX280#Temperature=25.93 3259063 : Info : EVENT: BMX280#Humidity=0.00 3259261 : Info : EVENT: BMX280#Pressure=982.20 sleep disable

      3262100 : Info  : WIFI : Connecting MyTestAP attempt #176

scandone 3265059 : Info : EVENT: WiFi#Disconnected 3265280 : Info : WIFI : Disconnected! Reason: '(201) No AP found' Connected for 2855 ms 3265281 : Info : WIFI : Set WiFi to OFFbcn 0 del if1 del if0 usl mode : null force slp enable,type: 2 fpm open,type:2 0

3266295 : Info : EVENT: WiFi#APmodeDisabled Fatal exception 0(IllegalInstructionCause): epc1=0x23524920, epc2=0x00000000, epc3=0x402a7d97, excvaddr=0x00000000, depc=0x00000000

                                              Exception (0):
                                                            epc1=0x23524920 epc2=0x00000000 epc3=0x402a7d97 excvaddr=0x00000000 depc=0x00000000

                                                               >>>stack>>>

                                                                          ctx: sys
  sp: 3ffffbf0 end: 3fffffb0 offset: 01a0
                                         3ffffd90:  401032c8 3fffc200 00000022 3fffc278
         3ffffda0:  402d1068 00000030 0000001c 00000000
                                                         3ffffdb0:  3fff37ec 3fff37b8 3fffa55c 402a4542
                         3ffffdc0:  00000003 3fff37b8 3fffa55c 402a4958
                                                                         3ffffdd0:  3ffe9c31 00000000 00000020 40100874
                                         3ffffde0:  00000006 00000006 00000000 3fffa5ac
         3ffffdf0:  3fffa55c 3ffe9cac 3fff37b8 402ab3b5
                                                         3ffffe00:  00000000 00000000 00000020 40100874
                         3ffffe10:  00000014 3fffa55c 3ffe9cac 402abe5c
                                                                         3ffffe20:  3ffea2dc 40101cb6 04000000 00000190
                                         3ffffe30:  3ffe9cb0 000000ff 00000000 00000011
         3ffffe40:  3fff3794 00000000 00000000 401032aa
                                                         3ffffe50:  00000011 00000001 3fff7d84 3fffa55c
                         3ffffe60:  3ffe9cac 3fffa55c 3fff3da4 402abeee
                                                                         3ffffe70:  3fff37b8 00000000 00000000 402a4a3f
                                         3ffffe80:  4000050c 3fffc200 00000022 402a87b8
         3ffffe90:  3fff37b8 3fff7d84 3fff7d84 402a9a87
                                                         3ffffea0:  3fff37b8 3ffe9cb0 00000043 3fffa5c0
                         3ffffeb0:  402dc4f4 3fffa6b8 3fff7d84 00000005
                                                                         3ffffec0:  3fffa55c 3fffa6b8 3fff7d84 402a9c67
                                         3ffffed0:  0000000e 0000000d 0000000d 401015d0
         3ffffee0:  3fff37b8 00000000 4bc6a7f0 402dc4ec
                                                         3ffffef0:  402a6330 0031d760 00000020 402dc4ec
                         3fffff00:  00000000 00000000 3fff37b8 402aac06
                                                                         3fffff10:  402a6330 0031d760 3fff3648 402a6340
                                         3fffff20:  402dc4ec 00000000 00000001 402a4b75
         3fffff30:  c2b1501d 3ffef5e0 3ffef5b8 402a64c4
                                                         3fffff40:  402c3cd8 3ffef5b8 3ffef5e0 60000600
                         3fffff50:  c2b1501d 3ffef5e0 3ffef5b8 402c3ce5
                                                                         3fffff60:  402c3d2a 3fffdab0 00000000 3fffdcb0
                                         3fffff70:  3ffef608 3fffdad0 3fff3058 40289d3f
         3fffff80:  40000f49 40000f65 3fffdab0 40000f49
                                                         3fffff90:  40000e19 40001878 00000002 3fff3010
                         3fffffa0:  3fffff10 40105725 40001878 00000002
                                                                         <<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 tail 8 chksum 0x2d csum 0x2d v643ec203 ~ld ▒"l`r▒olph▒U68 : Info :

INIT : Booting version: (ESP82xx Core 2_6_0, NONOS SDK 2.2.2-dev(bb83b9b), LWIP: 2.1.2 PUYA support) 69 : Info : INIT : Free RAM:29632 70 : Info : INIT : Warm boot #4 Last Task: Const Interval timer, id: 8 - Restart Reason: Exception 72 : Info : FS : Mounting... 97 : Info : FS : Mount successful, used 78061 bytes of 957314 561 : Info : CRC : program checksum ...OK 597 : Info : CRC : SecuritySettings CRC ...OK 625 : Info : INIT : Free RAM:26472 627 : Info : INIT : I2C 627 : Info : INIT : SPI not enabled 726 : Info : PN532: Found chip PN532 FW: 1.6 2404 : Info : INFO : Plugins: 79 [Normal] [Testing] (ESP82xx Core 2_6_0, NONOS SDK 2.2.2-dev(bb83b9b), LWIP: 2.1.2 PUYA support) 2405 : Info : EVENT: System#Wake ....

ghtester commented 4 years ago

FZYI now I experienced a node hung - it responds to PING, inaccessible web GUI, serial console stopped logging and it's frozen (latest Info: Uptime 116 ConnectFailures 22 FreeMem 11288 WiFiStatus 3)

Edit1: After about 10 minutes the node recovered without reboot... :-) Edit 2: After another 30 minutes node still visible also as AP while the WiFi signal to primary AP is good (current RSSI -72)...

TD-er commented 4 years ago

Can you show me the timingstats of that node, after it recovered?

Please keep in mind that every time you load that page the stats are reset, so no refresh if loading takes a while.

ghtester commented 4 years ago

Sorry, NOPE - rebooted many times in the meantime, tested many other things like pcfgpio (never tested before, works fine but after node reboot the first output command clears all other output pins - I think it was discussed somewhere a long time ago). But as far as I remember, at least the Uptime was also frozen (so after operation recovery it did not count these 10 minutes).

TD-er commented 4 years ago

Hmm, that's really strange. The Watchdog should have kicked in after 6 - 8 seconds. That's what the watchdog is for.

Do you have any plugin active that may be using an interrupt function for quick reaction to GPIO state change? Also the DHT thermometer sensors need disabled interrupts for proper reading.

ghtester commented 4 years ago

Well, I don't know. Maybe the iButton was enabled but I am not sure. plugins

TD-er commented 4 years ago

I don't see an active plugin which should be read using interrupts. If the config was like this since boot, then no interrupt should be defined. If a plugin was disabled since boot, then it might be the plugin is not performing a clean-up of those registered interrupts

ghtester commented 4 years ago

Just FYI... Can't reconnect as -2 networks found... :-)

173914815 : Info : EVENT: Clock#Time=Fri,20:16 173916084 : Info : WD : Uptime 2898 ConnectFailures 32 FreeMem 12752 WiFiStatus 6 sleep disable 173919413 : Info : WIFI : Connecting MyTestAP attempt #6001 173920901 : Info : BMP280 : Address: 0x76 173920902 : Info : BMP280 : Temperature: 23.95 173920902 : Info : BMP280 : Barometric Pressure: 982.50 173920903 : Info : EVENT: BMX280#Temperature=23.95 173921105 : Info : EVENT: BMX280#Humidity=0.00 173921303 : Info : EVENT: BMX280#Pressure=982.50 173937695 : Info : BH1750 Address: 0x23 Mode: 0x1 : Light intensity: 6.67 173937697 : Info : EVENT: Light#Lux=6.67 173939324 : Info : BMP280 : Address: 0x76 173939324 : Info : BMP280 : Temperature: 23.94 173939325 : Info : BMP280 : Barometric Pressure: 982.48 173939326 : Info : EVENT: BMX280#Temperature=23.94 173939528 : Info : EVENT: BMX280#Humidity=0.00 173939727 : Info : EVENT: BMX280#Pressure=982.48 sleep disable 173940076 : Info : WIFI : Connecting MyBackupAP attempt #6002 173944677 : Info : Dummy: value 1: 0.00 173944678 : Info : EVENT: Time#1=0.00 173944877 : Info : EVENT: Time#2=0.00 173945073 : Info : EVENT: Time#3=0.00 173945268 : Info : EVENT: Time#4=0.00 173946637 : Info : WD : Uptime 2899 ConnectFailures 32 FreeMem 12752 WiFiStatus 6

wifiscan 173958107 : Info : Command: wifiscan WIFI : SSID Scan start WIFI : -2 networks found

Ok sleep disable 173960203 : Info : WIFI : Connecting MyBackupAP attempt #6003 173961813 : Info : BMP280 : Address: 0x76 173961813 : Info : BMP280 : Temperature: 23.95 173961814 : Info : BMP280 : Barometric Pressure: 982.48 173961815 : Info : EVENT: BMX280#Temperature=23.95 173962015 : Info : EVENT: BMX280#Humidity=0.00 173962214 : Info : EVENT: BMX280#Pressure=982.48

After warm reboot invoked by reboot command from serial console the node reconnected.

Edit: compiled a custom build (including IRRX) in Vagrant using current sources, updated a node with this firmware to do testing during weekend...

Firmware

Build:⋄ | 20104 - Mega System Libraries:⋄ | ESP82xx Core 2_6_0, NONOS SDK 2.2.2-dev(bb83b9b), LWIP: 2.1.2 PUYA support Git Build:⋄ | My Build: Nov 15 201919:59:07 Plugins:⋄ | 37 [Normal] Build Md5: | 288b35a22146746e8f114f7966583165 Md5 check: | passed. Build Time:⋄ | Nov 15 2019 20:00:35 Binary Filename:⋄ | ESP_Easy_20191115_vagrant_custom_ESP8266_4M1M.bin

ghtester commented 4 years ago

Although it looks better, I am still experiencing the node reboot due to exhausted memory after many MQTT reconnections. Also there is an issue to WiFi reconnect after a warm ESP node reboot when the WiFi reception is weak - described here: #2721

TD-er commented 4 years ago

It is very well possible, the issues in PubSubClient are not the only causes of this issue.

ghtester commented 4 years ago

Yeah but if the WiFi signal is good, everything looks working pretty fine so for most users this issue does not exist.

ghtester commented 4 years ago

Weekend test result with RSSI about -80 dB and with MQTT Retain Msg: option turned off: 153844513 : Info : WD : Uptime 2564 ConnectFailures 84 FreeMem 7104 WiFiStatus 3 The main issue now is that the node can't reconnect to WiFi AP after the warm reboot when the WiFi signal is weak (reproducible by issuing the REBOOT command from serial console if RSSI is about -90). It looks that AP+STA mode is activated too quickly, thus the WiFi sensitivity is reduced and the ESP node can't connect to WiFi AP with weak signal anymore. Am I able to disable the AP mode completely somehow in custom Vagrant builds please? I really don't want already configured ESP nodes to switch to AP mode ever.

TD-er commented 4 years ago

I think the AP mode issue is different.

The first line of `ESPEasyWifi.ino:

#define WIFI_RECONNECT_WAIT                20000  // in milliSeconds

Just set it longer. Maybe 20 seconds is too short for the first attempt?

ghtester commented 4 years ago

Well, thanks for the hint. I don't know but what I see is a difference between cold boot and warm boot (reboot). As in my site there are many WiFi APs, after cold boot the WIFISCAN command does show more than 20 APs, after reboot only about 8 APs is visible from ESP node...

TD-er commented 4 years ago

I don't know if there is a difference in the core library for scanning from a cold boot or warm boot. I know I don't change behavior, although I could by storing the BSSID and channel in RTC to make the reconnect from warm boot really quick.

But still there is an issue regarding the AP mode activated, even after some time running, so there is a problem in that logic for sure.

ghtester commented 4 years ago

Yes it would be nice but I am afraid even if you try to reconnect with a known last BSSID & channel, after warm boot the sensitivity is decreased (I really wonder why) so the node can't see the AP with weak signal anymore...

ghtester commented 4 years ago

FYI, unfortunately the topic issue still persists, at least on my recent custom build:

Firmware

Build:⋄ | 20104 - Mega System Libraries:⋄ | ESP82xx Core 2_6_1, NONOS SDK 3.0.0-dev(c0f7b44), LWIP: 2.1.2 PUYA support Git Build:⋄ | My Build: Nov 21 201917:32:57 Plugins:⋄ | 37 [Normal] Build Md5: | e2d52b9dca1ae3c9e7ca431e929220 Md5 check: | passed. Build Time:⋄ | Nov 21 2019 17:34:24 Binary Filename:⋄ | ESP_Easy_20191121_vagrant_custom_sdk3_ESP8266_4M1M.bin

Now there're a more recent source code releases and the sdk 3.0.0 (which helped with a bad WiFi sensitivity issue after a warm boot) was discontinued so I'll try to make another custom build to test soon.

ghtester commented 4 years ago

I have found when I disable the MQTT Controller, CunnectFailures counter does not increase under the same conditions (RSSI about -90). So it looks that WiFi connectivity to AP is somehow stable (yes there may be some delays in communication).

5434802 : Info : WD : Uptime 91 ConnectFailures 0 FreeMem 11808 WiFiStatus 3

I thought this counter is related to WiFi status change.
As soon as I enable the MQTT Controller, the ConnectFailures counter is incremented (probably) when a timeout occurs and it happens quite often in my environment. Should it work this way that MQTT Controller is updating the same counter? I suppose it can also activate some WiFi related tasks and I am not sure if it's OK...

TD-er commented 4 years ago

ConnectFailures is a global counter which is incremented for any connection attempt made. Typically a controller is the one that makes connections to the outside world and MQTT controllers are the most active ones among the controllers we have.

So yes, the MQTT controller is also increasing this counter when it fails to connect to the broker.

ghtester commented 4 years ago

OK, thanks for the explanation. I tried to play with MQTT Controller settings but couldn't get the better results - the memory is still decreasing during a reconnect to MQTT broker. I tried to increase the timeout but in this case usually the Exception reason changes from 29 to 3.

TD-er commented 4 years ago

Exception reason changes from 29 to 3.

So it changed from WDT reboot to Stack Overflow. That's interesting.

TD-er commented 4 years ago

Can you test the test build I made last night for this PR: https://github.com/letscontrolit/ESPEasy/pull/2792#issuecomment-560276967

ghtester commented 4 years ago

OK, installed ESP_Easy_mega-20191130-3-PR_2792_test_beta_ESP8266_4M1M.bin to one node. The current status is following:

13083906 : Info : WD : Uptime 218 ConnectFailures 46 FreeMem 16376 WiFiStatus 6

This is after a warm reboot when the WiFi connectivity is significantly reduced perhaps due to SDK 2.2.2 bug. Although the WIFISCAN command can sometimes show my primary AP with a good RSSI level (about -76 dBm), the node can't connect in most attempts (currently over 4100 attempts).
I'll play with the WiFi AP signal level, it will take a long time to get some results...

TD-er commented 4 years ago

Hmm, so you have some setup with horrible WiFi and the unit does no longer crash and not run out of RAM? Or is it useless now?

ghtester commented 4 years ago

Well, please give me more time, the conditions mentioned above really made the ESP node useless (it was mostly disconnected). The ConnectFailures counter may say something (in past the node usually did not survive more than 40 reconnects) but some conditions also have changed in the meantime - there's now only one plugin (BMX280) sending data every 15 seconds. I have unshielded my AP a bit so now the signal is quite better. We'll see tomorrow how many reconnects are there and if the memory is still decreasing or not. Nevertheless, even the earlier beta firmware from 20191123 sources, core 2.7.0 and SDK 2.2.2 after a cold boot was quite stable even with weak WiFi signal. The SDK 2.2.2 WiFi bug after a warm boot is now the most annoying issue from my perspective.

ghtester commented 4 years ago

FYI the current status: 97233303 : Info : WD : Uptime 1621 ConnectFailures 426 FreeMem 14272 WiFiStatus 3 So it's not so bad... The memory is decreasing and increasing, the minimum I saw in Serial Console was about 12000, after boot it's about 16000. It needs to be tested with more plugins when the free RAM after a fresh boot is lower.

TD-er commented 4 years ago

I just merged 2 PRs of me which are related to both your issues, so you should be able to build yourself using Vagrant.

ghtester commented 4 years ago

Thanks a lot, Gijs, I'll give it a try. It's a pity there's an issue with SDK 2.2.2...

TD-er commented 4 years ago

Thanks a lot, Gijs, I'll give it a try. It's a pity there's an issue with SDK 2.2.2...

I also changed it to use core 2.6.2, which is almost the same as core 2.7.0 But you can also test with other SDK, as you now can list the envs to be built and I added a custom env with SDK3.

ghtester commented 4 years ago

Yes I know but when I recently tested the build with SDK3, it was not very stable in comparison to SDK2.2.2.

TD-er commented 4 years ago

I know, that's why I have not included it in the nightly builds. Let me know if you have more info, or maybe a link to other issues regarding the 'warm boot wifi reconnect' issues which may help finding out what's going on here. To me it sounds like a combination of timing critical stuff in combination with not the same values set as on cold boot.

ghtester commented 4 years ago

I have already compiled the custom Vagrant build and updated the node with ESP_Easy_20191203_vagrant_custom_sdk3_ESP8266_4M1M.bin, we'll see if there's a change with stability. Well, I don't know how the cold / warm node initialization works and what's the difference but it really looks like the different WiFi init parameters are used in cold / warm boot in SDK 2.2.2... I have no idea what could be the reason of degraded WiFi sensitivity after a warm boot and how to reproduce it after a cold boot (without REBOOT command or a crash due to exception). Everything else (in plugins I am using) seems to be working pretty fine to me in builds using SDK 2.2.2.