Pico W bootloop on WiFi connect

oddstr13 commented 9 months ago

The problem

Last line logged is [I][wifi:274]: WiFi Connecting to 'IoT'..., followed by a reboot a few seconds later. This happens more often than not, but after a few minutes it eventually manages to succeed (I counted 15 reboots once, haven't bothered counted other than that once).

Git bisect points to esphome/esphome#3509 being the breaking change, specifically, esphome/esphome@a9630ac847a292bc6ad671356804deb809eb6dc4 has the bootloop issue, while esphome/esphome@22c0b0ab does not.

Which version of ESPHome has the issue?

a9630ac -> dev

What type of installation are you using?

pip

Which version of Home Assistant has the issue?

No response

What platform are you using?

RP2040

Board

Custom, but problem also present for plain Pico W

Component causing the issue

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Discord thread; https://discord.com/channels/429907082951524364/1150959304589967461

hostcc commented 9 months ago

@oddstr13 , thanks for spotting that! I can 100% reproduce this, including bisected versions (https://github.com/esphome/esphome/commit/a9630ac847a292bc6ad671356804deb809eb6dc4 - bootloop, https://github.com/esphome/esphome/commit/22c0b0ab - ok) and affected version eventually would boot in unpredictable amount of reboots.

hostcc commented 9 months ago

Took another attempt to pinpoint the issue - using current dev (@11f6e555f9607767991a93d9c6259ef627a39bda) reverted all the changes over esphome/core introduced by the #pr3509:

$ git rev-parse HEAD
11f6e555f9607767991a93d9c6259ef627a39bda
$ git diff 22c0b0ab..a9630ac  esphome/core/ | patch -p1 -R
patching file 'esphome/core/__init__.py'
patching file 'esphome/core/config.py'
patching file 'esphome/core/defines.h'
patching file 'esphome/core/helpers.cpp'
patching file 'esphome/core/helpers.h'
patching file 'esphome/core/log.h'

Once compiled and uploaded the RP2040W boots fine, both after being reset during upload and after power on. So there is a change went to esphome/core introduced the issue - will attempt to investigate further.

Config is pretty minimal:

---
rp2040:
  board: rpipicow
  framework:
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

esphome:
  name: minimal

logger:
  level: VERY_VERBOSE

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_psk

hostcc commented 9 months ago

Sitrep: reverting just esphome/core/helpers.h fixes the issue:

$ git diff 22c0b0ab..a9630ac  esphome/core/helpers.h | patch -p1 -R
patching file 'esphome/core/helpers.h'

hostcc commented 9 months ago

Okay, following patch when applied over current dev seemingly fixes the issue. However, I don't understand why - the error I've added isn't raised by preprocessor (that is expected because USE_LIBRETINY should not be defined), while commenting includes out those should have been ignored for RP2040W makes a difference.

Not being very familiar with the toolchain, but it seems like either I'm not seeing something staring at me, or an issue in preprocessor (highly unlikely?), or a tool scans the source files and somehow trips on those includes should have been skipped (USE_LIBRETINY is no defined).

diff --git a/esphome/core/helpers.h b/esphome/core/helpers.h
index c3ed443b..8bd695a7 100644
--- a/esphome/core/helpers.h
+++ b/esphome/core/helpers.h
@@ -18,8 +18,9 @@
 #include <freertos/FreeRTOS.h>
 #include <freertos/semphr.h>
 #elif defined(USE_LIBRETINY)
-#include <FreeRTOS.h>
-#include <semphr.h>
+#error libretiny
+// #include <FreeRTOS.h>
+// #include <semphr.h>
 #endif

 #define HOT __attribute__((hot))

oddstr13 commented 9 months ago

I can confirm that

commenting out those two includes does indeed seem to fix the issue
the error preprocessor statements do not fire inside the elif
and that #error correctly aborts the compilation when placed outside the if block

wtf?

hostcc commented 9 months ago

That is LDF (of PIO) acting wrongly detecting FreeRTOS as dependency:

Original dev:

Dependency Graph
|-- WiFi @ 1.0.0
|-- LEAmDNS @ 1.2
|-- lwIP-Ethernet @ 1
|-- FreeRTOS @ 1.0.0

Patch applied to esphome/core/helpers.h:

Dependency Graph
|-- WiFi @ 1.0.0
|-- LEAmDNS @ 1.2
|-- lwIP-Ethernet @ 1

There are multiple issues related to CPP emulation in PIO. Will have another look to see if there would be a workaround

hostcc commented 9 months ago

Well, going to conclude LDF works as designed, since it runs in chain mode that

Does not evaluate C/C++ Preprocessor conditional syntax

(from esphome -v compile ...):

...
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
...

Going to try chain+ instead and see how that goes

oddstr13 commented 9 months ago

…amazed it works at all then, esphome is nothing but ifdefs

https://github.com/platformio/platformio-core/issues/3113#issuecomment-1704311265 suggests deep mode, and the follow up commend suggests that using lib_deps could be an alternative – which is in use here, maybe except from the new libretiny stuff? I don't know how inheritance works in the pio config.

hostcc commented 9 months ago

Did a try with chain+ mode instead (since it should be similar to chain used by default) - doesn't include FreeRTOS as discovered dependency and resulting firmware boots ok. However I'm not being very comfortable with such broad change. Interestingly, ESP8266, Libretiny and other platforms use off for LDF mode but attempt using it for RP2040 resulted in missing dependencies - I might be exploring that further.

diff --git a/esphome/components/rp2040/__init__.py b/esphome/components/rp2040/__init__.py
index b31192f7..9dcdb2ca 100644
--- a/esphome/components/rp2040/__init__.py
+++ b/esphome/components/rp2040/__init__.py
@@ -152,6 +152,7 @@ CONFIG_SCHEMA = cv.All(
 async def to_code(config):
     cg.add(rp2040_ns.setup_preferences())

+    cg.add_platformio_option("lib_ldf_mode", "chain+")
     cg.add_platformio_option("board", config[CONF_BOARD])
     cg.add_build_flag("-DUSE_RP2040")
     cg.add_define("ESPHOME_BOARD", config[CONF_BOARD])

Results in seemingly correct dependencies:

...
Dependency Graph
|-- WiFi @ 1.0.0
|-- LEAmDNS @ 1.2
|-- lwIP-Ethernet @ 1
|-- Updater @ 1.0
|-- MD5Builder @ 1.0.0
...

hostcc commented 9 months ago

Adding @jesserockz and @kbx81 for vis.

oddstr13 commented 9 months ago

The RP2040 platform doesn't really define any dependencies it looks like? https://github.com/esphome/esphome/blob/dadbc1aefa267d1868aa189da6a3ee16bb318674/platformio.ini#L165L166

hostcc commented 9 months ago

The RP2040 platform doesn't really define any dependencies it looks like? https://github.com/esphome/esphome/blob/dadbc1aefa267d1868aa189da6a3ee16bb318674/platformio.ini#L165L166

I think it should pull deps from common:arduino profile, but need to poke around more

oddstr13 commented 9 months ago

Confirming dev now works as expected

esphome / issues