Open jflecool2 opened 2 years ago
I found people with identical issues. Hope this helps the investigation!! https://community.home-assistant.io/t/nspanel-esp-stuck-in-boot-after-adding-working-component/394634/6
Went on Dev build of ESPHome. As of right now, dev build doesn't work either, however OTA/ping also stops working, which is worse...
I see the same behaviour: Everything was working fine, then I added a new item (platform: homeassistant, but I think this doesn't matter) and now I cannot connect the epshome anymore and I need to reupload the firware using Serial Connection.
Did you try to put boot ESPHome in Safe Mode before doing OTA? If you use memory intensive stuff like BLE, OTA is likely to crash... Safe Mode helps with that...
I ordered additional hardware to test while having serial logging and easy flashing. Can you confirm TXD0 should output logs once I power it up without GPIO0 grounded? Is there any config I can apply to have more verbose output? I would expect memory intensive stuff to impact boot & loop speed, not breaking OTA+iresponsive...
I received the hardware and took a peek at the generated cpp code and I don't have to go far to find a very possible issue: 100% of the objects are allocated dynamically. So if the memory is full, new will return 0, and the following call will be a Null Pointer Exception. I dont have proof that this is whats happening, but one can be certain that this will eventually occur. We could check if the pointer is null and return; if so, leaving some item uninitialized and raising a log message. However, I firmly believe it should be statically allocated so platformio see that we exceeded the board limit and refuse to proceed. However, we are limited by constructor. This limitation can be lifted by the Arduino-compatible solution given by jfpoilpret here. Once remote bricking is not a thing anymore, I'll try to find ways to make footprints of my objects smaller :)
Did an edit to cpp_generator.py and my image jumped from 42k to 66k. It runs on my WT32-ETH01 just fine! Grep'ping "= new" and there are other stuff, but not enough to fill out the memory, assuming I trust partitions.csv. I'll try to find time to mod my new NSPanel to try to reproduce, with serial connected. Maybe the output will be useful ...
I encountered a similar problem with a Nextion Intelligent display and ESP32 D1mini. If I add a switch platform:nextion to my yaml the esp goes into continuous reboot, 6 switch ok - 7 switch fail. No log from Esphome but from serial with Esphome Flasher I have this type of log:
[22:17:07]Rebooting... [22:17:07]ets Jul 29 2019 12:21:46 [22:17:07] [22:17:07]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [22:17:07]configsip: 0, SPIWP:0xee [22:17:07]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [22:17:07]mode:DIO, clock div:2 [22:17:07]load:0x3fff0018,len:4 [22:17:07]load:0x3fff001c,len:952 [22:17:07]load:0x40078000,len:6084 [22:17:07]load:0x40080000,len:7944 [22:17:07]entry 0x40080310 [22:17:08]Guru Meditation Error: Core 1 panic'ed (Unhandled debug exception) [22:17:08]Debug exception reason: Stack canary watchpoint triggered (loopTask) [22:17:08]Core 1 register dump: [22:17:08]PC : 0x40086f94 PS : 0x00060e36 A0 : 0x3ffb0120 A1 : 0x3ffb0060 [22:17:08]A2 : 0x00000001 A3 : 0x3ffc8130 A4 : 0x3ffc8130 A5 : 0x00000001 [22:17:08]A6 : 0x00060e20 A7 : 0x00000000 A8 : 0x800855c8 A9 : 0x3ffb0100 [22:17:08]A10 : 0x3ff000e0 A11 : 0x00000001 A12 : 0x3ffbe8e8 A13 : 0x00000001 [22:17:08]A14 : 0x00060e23 A15 : 0x00000000 SAR : 0x0000000e EXCCAUSE: 0x00000001 [22:17:08]EXCVADDR: 0x00000000 LBEG : 0x40001609 LEND : 0x4000160d LCOUNT : 0x00000000 [22:17:08] [22:17:08]ELF file SHA256: 0000000000000000 [22:17:08] [22:17:08]Backtrace: 0x40086f94:0x3ffb0060 0x3ffb011d:0x3ffb0140 0x40172b43:0x3ffb0180 0x40172c1a:0x3ffb01a0 0x400878e2:0x3ffb01c0 0x40087de5:0x3ffb01e0 0x4017a595:0x3ffb0250 0x40178e55:0x3ffb0270 0x40179287:0x3ffb0290 0x40177c5f:0x3ffb0300 0x40178041:0x3ffb0350 0x4017816f:0x3ffb03d0 0x40177145:0x3ffb0430 0x40177791:0x3ffb0470 0x4014a52a:0x3ffb0490 0x4014a692:0x3ffb04d0 0x4014a6b9:0x3ffb04f0 0x40108797:0x3ffb0510 0x401088af:0x3ffb0530 0x40170c49:0x3ffb0550 0x400fa01a:0x3ffb0570 0x400e7bfd:0x3ffb0680 0x400e7c5c:0x3ffb06b0 0x400e6b2d:0x3ffb06e0 0x401a4df5:0x3ffb07f0 0x401a4ec1:0x3ffb0810 0x400e992d:0x3ffb0830 0x400e0ad5:0x3ffb0880 0x400ef14e:0x3ffb08c0 0x40106706:0x3ffb1fb0 0x40089bb2:0x3ffb1fd0 [22:17:08] [22:17:08]Rebooting... [22:17:08]ets Jul 29 2019 12:21:46 [22:17:08] [22:17:08]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [22:17:08]configsip: 0, SPIWP:0xee [22:17:08]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [22:17:08]mode:DIO, clock div:2 [22:17:08]load:0x3fff0018,len:4 [22:17:08]load:0x3fff001c,len:952 [22:17:08]load:0x40078000,len:6084 [22:17:08]load:0x40080000,len:7944 [22:17:08]entry 0x40080310 [22:17:10]bkcmd=0sleep=0bkcmd=0sleep=0restGuru Meditation Error: Core 1 panic'ed (Unhandled debug exception) [22:17:10]Debug exception reason: Stack canary watchpoint triggered (loopTask) [22:17:10]Core 1 register dump: [22:17:10]PC : 0x40086f99 PS : 0x00060e36 A0 : 0x00060e30 A1 : 0x3ffb0070 [22:17:10]A2 : 0x00000001 A3 : 0x3ffc8130 A4 : 0x3ffc8130 A5 : 0x00000001 [22:17:10]A6 : 0x00060e20 A7 : 0x00000000 A8 : 0x800855c8 A9 : 0x3ffb0110 [22:17:10]A10 : 0x3ff000e0 A11 : 0x00000001 A12 : 0x3ffbe8e8 A13 : 0x00000001 [22:17:10]A14 : 0x00060e23 A15 : 0x00000000 SAR : 0x0000000a EXCCAUSE: 0x00000001 [22:17:10]EXCVADDR: 0x00000000 LBEG : 0x40001609 LEND : 0x4000160d LCOUNT : 0x00000000 [22:17:10] [22:17:10]ELF file SHA256: 0000000000000000 [22:17:10] [22:17:10]Backtrace: 0x40086f99:0x3ffb0070 0x00060e2d:0x3ffb0150 0x40172b43:0x3ffb0190 0x40172c1a:0x3ffb01b0 0x400878e2:0x3ffb01d0 0x40087de5:0x3ffb01f0 0x4017a595:0x3ffb0260 0x40178e55:0x3ffb0280 0x40179287:0x3ffb02a0 0x4017941e:0x3ffb0310 0x401780c0:0x3ffb0390 0x4017816f:0x3ffb0410 0x40177145:0x3ffb0470 0x40177791:0x3ffb04b0 0x4014a52a:0x3ffb04d0 0x4014a692:0x3ffb0510 0x4014a6b9:0x3ffb0530 0x40108797:0x3ffb0550 0x401088af:0x3ffb0570 0x40170c49:0x3ffb0590 0x400fa01a:0x3ffb05b0 0x400e7bfd:0x3ffb06c0 0x400e7c5c:0x3ffb06f0 0x400e6b2d:0x3ffb0720 0x401a4df5:0x3ffb0830 0x401a4ec1:0x3ffb0850 0x400e992d:0x3ffb0870 0x400f8382:0x3ffb08c0 0x40106706:0x3ffb1fb0 0x40089bb2:0x3ffb1fd0 [22:17:10] [22:17:10]Rebooting... [22:17:10]ets Jul 29 2019 12:21:46 [22:17:10] [22:17:10]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [22:17:10]configsip: 0, SPIWP:0xee [22:17:10]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [22:17:10]mode:DIO, clock div:2 [22:17:10]load:0x3fff0018,len:4 [22:17:10]load:0x3fff001c,len:952 [22:17:10]load:0x40078000,len:6084 [22:17:10]load:0x40080000,len:7944 [22:17:10]entry 0x40080310 [22:17:11]bkcmd=0sleep=0bkcmd=0sleep=0restGuru Meditation Error: Core 1 panic'ed (Unhandled debug exception) [22:17:11]Debug exception reason: Stack canary watchpoint triggered (loopTask) [22:17:11]Core 1 register dump: [22:17:11]PC : 0x40086f99 PS : 0x00060e36 A0 : 0x00060e30 A1 : 0x3ffb0070 [22:17:11]A2 : 0x00000001 A3 : 0x3ffc8130 A4 : 0x3ffc8130 A5 : 0x00000001 [22:17:11]A6 : 0x00060e20 A7 : 0x00000000 A8 : 0x800855c8 A9 : 0x3ffb0110 [22:17:11]A10 : 0x3ff000e0 A11 : 0x00000001 A12 : 0x3ffbe8e8 A13 : 0x00000001 [22:17:11]A14 : 0x00060e23 A15 : 0x00000000 SAR : 0x0000000a EXCCAUSE: 0x00000001 [22:17:11]EXCVADDR: 0x00000000 LBEG : 0x40001609 LEND : 0x4000160d LCOUNT : 0x00000000 [22:17:11] [22:17:12]ELF file SHA256: 0000000000000000 [22:17:12] [22:17:12]Backtrace: 0x40086f99:0x3ffb0070 0x00060e2d:0x3ffb0150 0x40172b43:0x3ffb0190 0x40172c1a:0x3ffb01b0 0x400878e2:0x3ffb01d0 0x40087de5:0x3ffb01f0 0x4017a595:0x3ffb0260 0x40178e55:0x3ffb0280 0x40179287:0x3ffb02a0 0x4017941e:0x3ffb0310 0x401780c0:0x3ffb0390 0x4017816f:0x3ffb0410 0x40177145:0x3ffb0470 0x40177791:0x3ffb04b0 0x4014a52a:0x3ffb04d0 0x4014a692:0x3ffb0510 0x4014a6b9:0x3ffb0530 0x40108797:0x3ffb0550 0x401088af:0x3ffb0570 0x40170c49:0x3ffb0590 0x400fa01a:0x3ffb05b0 0x400e7bfd:0x3ffb06c0 0x400e7c5c:0x3ffb06f0 0x400e6b2d:0x3ffb0720 0x401a4df5:0x3ffb0830 0x401a4ec1:0x3ffb0850 0x400e992d:0x3ffb0870 0x400f8382:0x3ffb08c0 0x40106706:0x3ffb1fb0 0x40089bb2:0x3ffb1fd0 [22:17:12] [22:17:12]Rebooting... [22:17:12]ets Jul 29 2019 12:21:46 [22:17:12] [22:17:12]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) [22:17:12]configsip: 0, SPIWP:0xee [22:17:12]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 [22:17:12]mode:DIO, clock div:2 [22:17:12]load:0x3fff0018,len:4 [22:17:12]load:0x3fff001c,len:952 [22:17:12]load:0x40078000,len:6084 [22:17:12]load:0x40080000,len:7944 [22:17:12]entry 0x40080310
Moreover, in order to reflect from usb I absolutely have to completely disconnect the display otherwise I have this log
Using '/dev/cu.usbserial-025968DB' as serial port. Connecting...[22:32:18]ets Jul 29 2019 12:21:46 [22:32:18] [22:32:18]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:18]waiting for download .....[22:32:20]ets Jul 29 2019 12:21:46 [22:32:20] [22:32:20]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:20]waiting for download _____[22:32:22]ets Jul 29 2019 12:21:46 [22:32:22] [22:32:22]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:22]waiting for download .....[22:32:24]ets Jul 29 2019 12:21:46 [22:32:24] [22:32:24]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:24]waiting for download _____[22:32:25]ets Jul 29 2019 12:21:46 [22:32:25] [22:32:25]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:25]waiting for download .....[22:32:27]ets Jul 29 2019 12:21:46 [22:32:27] [22:32:27]rst:0x1 (POWERON_RESET),boot:0x3 (DOWNLOAD_BOOT(UART0/UART1/SDIO_REI_REO_V2)) [22:32:27]waiting for download __Serial port closed!
I bought a new nspanel. I took my working ESPHome config, flashed, and HA offered me to add it (I declined). Everything seems to work. Then I added a bunch of silly objects:
Then flashed it (again, by Serial/UART). Bootloop! Until watchdog take over (Last Boot was an unhandled reset, will proceed to safe mode in 0 restart)
Here's the log over Serial: nspanel too much entity bug serial.txt
I bought a new nspanel. I took my working ESPHome config, flashed, and HA offered me to add it (I declined). Everything seems to work. Then I added a bunch of silly objects:
- platform: nextion name: $device_name limit13 internal: true page_id: 10 component_id: 13
Then flashed it (again, by Serial/UART). Bootloop! Until watchdog take over (Last Boot was an unhandled reset, will proceed to safe mode in 0 restart)
Here's the log over Serial: nspanel too much entity bug serial.txt
This seems to confirm what I suspect as well. In my case everything works if I comment out 1 sensor. It doesn't matter which sensor I comment out. With all sensors it boot loops. This makes me think that I am running into some limit even though the build shows my storage and memory are fine.
On 2022.5.0 safe mode didn't work but after updating to 2022.5.1 they now enter safe mode and I can reflash OTA. So that is an improvement.
Do you have a safe mode button/switch configured? If you press it, will it reboot in safe mode and stay like that immediately?
Maybe related: I often see that if I press the safe mode button it won't reboot in safe, it will reboot normal. If I press safe mode button immediately again, it will stick in safe boot.
I have the same problem...
Just adding one cover command to a button make the nspanel no more available (web/ping etc) but wireless reflash ok. Adding more make it freeze completely and need an usb /serial flash again.
Didi you find something to avoid that?
trying to pass it in safe mode before didn't change anyting.
this doesn't work
This one is ok.
I can change the cover button removing one/adding another and vice/versa but not put both at the same time.
Does the bin generated file size is in case? Do i need to find another solution to handle all my cover ;(
I solve the issue with both action:
esphome:
name: nspanelsdb1
comment: $device_name
platformio_options:
build_flags:
- -DCONFIG_ARDUINO_LOOP_STACK_SIZE=32768
and
# Functionality for the Nextion display
external_components:
- source: github://pr#2956
components: [nextion]
refresh: 1h
- source: github://epiclabs-io/esphome@fix-885-old
components: wifi
refresh: 1h
Adding those two nex option and the wifi lib make me able to had all my cover and other button for now.
@jflecool2 could you try this ? https://github.com/esphome/esphome/pull/3577#issuecomment-1159689900
@tikismoke pr#2956 doesnt seem to be related to this issue. From what I see it changes from static to pointer the network object? CONFIG_ARDUINO_LOOP_STACK_SIZE=32768 will only delay the issue. Maybe I will be able to use 100 more objects. Maybe 10. Or maybe my device wont have the memory to handle this stack size. TBD. I still think ESPhome should redefine the New operator and if no space is available, reboot to safe mode or something of the kind. The code is filled with dynamic objects that assumes memory is infinite and always available. Anyway, I chose the path of least resistance: I made a generic sensor. All my buttons on NSPanel change the sensor to a number. ESPHome only task is to, on on_raw_value, send this to HA script "general purpose signal" which make different action depending on the number. That way I can have a huge number of action and esphome will never breaks! I don't even need to re-flash ESPHome to add new button on NSPanel! So I will leave this bug open, in a hope someone (or me if I become more familiar with esphome codebase someday) actually handles the dynamic memory adequately, either by banning dynamic memory or handling exceptions better. However, thank you for sending me these suggestions. It is appreciated!!
@tikismoke pr#2956 doesnt seem to be related to this issue.
No it's the wifi option that solve completely the issue for me. I have add since ~ 15/20 devices/sensors to the nspanel since.
Could we get detail about your solution with only on sensor for all button it's interesting 🤔
Home Assistant script "gpsig":
alias: gpsig
sequence:
- choose:
- conditions:
- condition: template
value_template: "{{ actionid == \"1\" }}"
sequence:
- if:
- condition: template
value_template: "{{ argstr1 == \"1234\" }}"
then:
- service: script.alarm_lock_disarm
data: {}
else:
- service: notify.mobile_app_nexus4
data:
message: Someone tried to unlock the alarm with {{ argstr1 }}.
- conditions:
- condition: template
value_template: "{{ actionid == \"2\" }}"
sequence:
- service: script.alarm_lock
data: {}
default: []
mode: single
icon: mdi:robot-industrial
fields:
actionid:
name: actionid
description: actionid
selector:
number:
min: 0
max: 100000
step: 1
argstr1:
name: argstr1
description: argstr1
selector:
text: null
argint1:
name: argint1
description: argint1
selector:
number:
min: 0
max: 100000
step: 1
argint2:
name: argint2
description: argint2
selector:
number:
min: 0
max: 100000
step: 1
ESPHome gpsig transfering (never use on_value, because we want triggering the same action to trigger it. 1->1 won't trigger on_value but will trigger on_raw_value):
text_sensor:
- platform: nextion
nextion_id: disp1
name: argstr1
id: argstr1
update_interval: 4s
component_name: argstr1
on_value:
- logger.log: argstr1 recvd
sensor:
- platform: nextion
internal: true
id: argint1
variable_name: argint1
- platform: nextion
internal: true
id: argint2
variable_name: argint2
- platform: nextion
internal: true
id: gpsig
variable_name: gpsig
on_raw_value:
- logger.log: 'gpsig up'
- homeassistant.service:
service: script.gpsig
data:
actionid: !lambda return (int)(x);
argstr1: !lambda return id(argstr1).state.c_str();
argint1: !lambda return (int)(id(argint1).state);
argint2: !lambda return (int)(id(argint2).state);
Nextion/NSPanel calling gpsig (user entered code in argstr1, action 1):
printh 92
prints "argstr1",0
printh 00
prints code.txt,0
printh 00
printh FF FF FF
printh 91
prints "gpsig",0
printh 00
prints 1,0
printh FF FF FF
code.txt=""
Hope it helps!
😱🤗 ok it's harder works template/script/automation for me 🤣.
I'll continue with the wifi patches that let the nspanel working as it should 👍
@jflecool2 This is @jpeletier, author of https://github.com/esphome/esphome/pull/3577
Can you try this and see if it solves your issue? I've been working with huge configs since then, without having to increase the stack size:
external_components:
- source: github://epiclabs-uc/esphome@fix-885-old
components: wifi
refresh: 0s
Check out the PR to understand the root cause. TL;DR; An ESP actually has memory to support very large configs, but some compiler quirks waste stack space, which is specially bad during component setup. The PR works around the issue by freeing earlier a temporary object during wifi configuration, which for some reason triggers that wasted stack bug.
If it solves the issue for you, please report in https://github.com/esphome/esphome/pull/3577 to support the PR. Thanks!
The problem
I bricked my NSPanel with OTA. Was able to reflash on it using headers. I added back my code gradually. I then found the breaking point. However, after playing with it for some time, I realized it didn't fail because of the thing itself, just too many things. I added 10 dummy binary sensor: fails. Commented out 9/10 of those: works. (limit1 ... limit10) When it got bricked, I couldn't ping it anymore. fully dead. However, when I am exceeding the limit by a small bit only, mDNS stops working, Logs doesnt work either. Button trigger doesnt trigger. However, OTA/ping still works.
RAM: [= ] 13.1% (used 42860 bytes from 327680 bytes) Flash: [====== ] 63.9% (used 1172816 bytes from 1835008 bytes)
I dont know what kind of buffer I am exceeding... To be fair, there's 60 "platform:" in my yaml. Not very small... I'll do my best to give additional information you require. https://imgur.com/a/tiwAAPR PS: Thanks for working on ESPHome. I hope one day I can contribute to it!Which version of ESPHome has the issue?
2022.3.1
What type of installation are you using?
Home Assistant Add-on
Which version of Home Assistant has the issue?
2022.3.8
What platform are you using?
ESP32
Board
NSPanel
Component causing the issue
Nextion or generic
Example YAML snippet
Anything in the logs that might be useful for us?
Additional information
No response