Open hmueller01 opened 10 months ago
After building another project with similar code and crashes I found out that this was the problem:
configTime(TZ_Europe_Berlin, PSTR("de.pool.ntp.org"), PSTR("europe.pool.ntp.org"), PSTR("time.nist.gov"));
Changing this to
configTime(TZ_Europe_Berlin, "de.pool.ntp.org", "europe.pool.ntp.org", "time.nist.gov");
did the trick.
But it didn't fully help on this project. There I still have a crash after a zalloc
/ malloc
right here:
void mqttCallback(char *topic, byte *payload, unsigned int length) {
bool topic_handled = false;
char topic_pub[100];
DEBUGF_P("%s" LF, __func__);
DEBUGF_P("getFreeHeap: %d, getHeapFragmentation: %d, getMaxFreeBlockSize: %d" LF, ESP.getFreeHeap(), ESP.getHeapFragmentation(), ESP.getMaxFreeBlockSize());
DEBUGF_P("length: %d" LF, length);
// copy payload to a char string
//char *data = (char *)zalloc(length + 1);
char *data = (char *)malloc(length + 1);
DEBUG_P("after malloc, before memcpy");
memcpy(data, payload, length);
It doesn't reach the DEBUG_P("after malloc, before memcpy");
any more.
But how is this possible? length
is only 2 bytes and the getMaxFreeBlockSize is 12080!
timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Jan 21 20:08:31 2024
mqttSetup: MQTT client id: ESP-0015AADA
mqttSetup: HomA system id: 123456-garage
mqttSetup: Attempting connection to server mqtt:8883
mqttSetup: Sucsessfully connected.
Stop multiResetDetecting
mqttConnect: connected using PubSubClient
cisternSwitch: switch cistern OFF
mqttConnect: Current time: Sun Jan 21 20:08:32 2024
mqttCallback
getFreeHeap: 13712, getHeapFragmentation: 12, getMaxFreeBlockSize: 12080
length: 2
No poison after block at: 0x3fff2838, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Panic umm_malloc.cpp:751 umm_malloc_core
sp: 3ffffcc0 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
ctx: cont
0x40101118: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x4010145a: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x401014c4: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_poison.c:173
0x40263737: umm_max_block_size at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_info.c:203
0x401008f8: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/heap.cpp:250
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x40208da5: mqttCallback(char*, unsigned char*, unsigned int) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:455
0x4021244b: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:497
0x40211d28: BearSSL::WiFiClientSecureCtx::_engineConnected() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:261 (discriminator 5)
0x40212c38: BearSSL::WiFiClientSecureCtx::read(unsigned char*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:383 (discriminator 4)
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x402295e8: BearSSL::WiFiClientSecureCtx::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:414
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x40212bbc: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x402289d2: BearSSL::WiFiClientSecure::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:279
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x4020d9c5: PubSubClient::readByte(unsigned char*) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:297
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x40228ea4: std::_Function_handler<void (char*, unsigned char*, unsigned int), void (*)(char*, unsigned char*, unsigned int)>::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x40100160: std::function<void (char*, unsigned char*, unsigned int)>::operator()(char*, unsigned char*, unsigned int) const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x4020e3f0: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:416
0x3fff0478: _Pri_3_Stack at nmi-vector.o:?
0x3ffef968: _Pri_3_Stack at nmi-vector.o:?
0x40222f28: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef968: _Pri_3_Stack at nmi-vector.o:?
0x402213f4: esp8266::MDNSImplementation::MDNSResponder::update() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS.cpp:1248
0x3ffef508: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:96
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffef508: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:96
0x4020d1d0: loop at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:937
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x4021a0c8: HardwareSerial::write(unsigned char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/HardwareSerial.h:194
0x4021b454: Print::printf_P(char const*, ...) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Print.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40210afe: void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:69
(inlined by) void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:78
(inlined by) ClientContext::connect(ip4_addr*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/include/ClientContext.h:147
(inlined by) WiFiClient::connect(IPAddress, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClient.cpp:164
0x40281178: etharp_output at ??:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40212a40: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:229
0x4022ac10: base64_encode_value at ??:?
0x4022ac10: base64_encode_value at ??:?
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x40281178: etharp_output at ??:?
0x3ffef4d8: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:98
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40208103: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:433
0x40228dfc: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x40206650: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x40209ca8: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:922
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:243
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x4021efb2: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:263
0x401005a9: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/cont.S:81
No poison after block at: 0x3fff2838, actual data: 0x65 0x0 0xa5 0xa5 Panic umm_malloc.cpp:751 umm_malloc_core
Memory is corrupted, allocator is configured to panic() when that happens
Tx for the quick response. Is there a way to figure out the location in the trace? I currently have no clue where to look. I already removed almost everything except the WiFi configuration and the mqtt stuff (over TLS using BearSSL on top of PubSubClient). This is really a pain ...
It doesn't reach the
DEBUG_P("after malloc, before memcpy");
any more.
Adding Serial.flush()
after every printf
might help to narrow the issue.
You can also try to use gdb.
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
...or something to do with SDK3? I don't see this one in the default edit: are you analogWrite'ing or running something in the NMI?
It doesn't reach the
DEBUG_P("after malloc, before memcpy");
any more.Adding
Serial.flush()
after everyprintf
might help to narrow the issue. You can also try to use gdb.
Thanks @d-a-v . It crashes exactly at the malloc function (added the flush after the malloc printf / DEBUG_P). Thanks for the gdb hint. Wasn't aware that this is possible with ESP8266 ... but if memory is corrupted I doubt gdb will be of much help. :-(
nmi-vector.o:?
I use the Ticker, but only once_scheduled() or attach_ms_scheduled() which should execute the callback in the loop context. I will remove all that code and will see if anything changes.
BTW: The crash happens in the PubSubClient MQTT callback message/topic received. But I use this code in other projects in the same way and do not see a crash there.
Edit: And for sure PubSubClient isn't bugfree - and unsupported since a long time. I already fixed a bug in context with PROGMEM data
boolean PubSubClient::publish_P(const char* topic, const char* payload, boolean retained) {
return publish_P(topic, (const uint8_t*)payload, payload ? strnlen(payload, this->bufferSize) : 0, retained);
}
to (as I have seen crashes at strnlen())
boolean PubSubClient::publish_P(const char* topic, const char* payload, boolean retained) {
return publish_P(topic, (const uint8_t*)payload, payload ? strnlen_P(payload, this->bufferSize) : 0, retained);
}
Removing all the Ticker code didn't help at all. Dump output looks almost the same.
--------------- CUT HERE FOR EXCEPTION DECODER ---------------
mqttConnect: connected using PubSubClient
mqttConnect: Current time: Mon Jan 22 20:29:47 2024
mqttCallback
getFreeHeap: 14872, getHeapFragmentation: 2, getMaxFreeBlockSize: 14712
length: 2
No poison after block at: 0x3fff2410, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Panic umm_malloc.cpp:751 umm_malloc_core
sp: 3ffffcc0 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
ctx: cont
0x40100ea0: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x401011e2: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x4010124c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_poison.c:173
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40261da3: umm_max_block_size at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_info.c:203
0x40100680: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/heap.cpp:250
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x4020c518: mqttCallback(char*, unsigned char*, unsigned int) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:451
0x40211e6b: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:497
0x40211748: BearSSL::WiFiClientSecureCtx::_engineConnected() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:261 (discriminator 5)
0x40212658: BearSSL::WiFiClientSecureCtx::read(unsigned char*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:383 (discriminator 4)
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x40227f20: BearSSL::WiFiClientSecureCtx::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:414
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x402125dc: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x402273ce: BearSSL::WiFiClientSecure::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:279
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4020d3e1: PubSubClient::readByte(unsigned char*) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:297
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4022784c: std::_Function_handler<void (char*, unsigned char*, unsigned int), void (*)(char*, unsigned char*, unsigned int)>::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x40100160: std::function<void (char*, unsigned char*, unsigned int)>::operator()(char*, unsigned char*, unsigned int) const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4020de0c: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:416
0x3fff0228: _Pri_3_Stack at nmi-vector.o:?
0x3ffef718: _Pri_3_Stack at nmi-vector.o:?
0x402223fc: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef718: _Pri_3_Stack at nmi-vector.o:?
0x402208c8: esp8266::MDNSImplementation::MDNSResponder::update() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS.cpp:1248
0x3ffef2c0: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:87
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffef2c0: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:87
0x4020cc18: loop at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:944
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40219590: HardwareSerial::write(unsigned char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/HardwareSerial.h:194
0x4021a91c: Print::printf_P(char const*, ...) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Print.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4021051e: void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:69
(inlined by) void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:78
(inlined by) ClientContext::connect(ip4_addr*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/include/ClientContext.h:147
(inlined by) WiFiClient::connect(IPAddress, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClient.cpp:164
0x4027f7c8: etharp_output at ??:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40212460: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:229
0x402294b8: base64_encode_value at ??:?
0x402294b8: base64_encode_value at ??:?
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x4027f7c8: etharp_output at ??:?
0x3ffef290: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020804f: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:429
0x402277d8: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4020664c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x3ffef838: _Pri_3_Stack at nmi-vector.o:?
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40209500: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:927
0x401003e0: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:243
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x4021e0e2: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:263
0x4010171d: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/cont.S:81
garage-arduino.cpp _Pri_3_Stack lines mentioned above:
86: ESP8266WebServer m_webserver(80);
87: ESP_WiFiManager_Lite m_wifi_manager;
88: WiFiEventHandler m_station_got_ip_handler;
89: BearSSL::WiFiClientSecure m_esp_client;
90: PubSubClient m_mqtt_client(m_esp_client);
garage-arduino.cpp _Pri_3_Stack lines mentioned above:
...I meant that this shows up b/c of enabled SDK3, apparently. SDK2 190703 does not have this symbol in .a, SDK305 does. platformio.ini options weren't mentioned above
@mcspr You should have access to https://github.com/arduino-hmueller01/garage-pio/blob/main/platformio.ini
Yes, I use SDK305. I'll see if it compiles with SDK2 and report.
Edit: I used [env:upstream_develop_flash_serial_debug] target here ...
https://github.com/arduino-hmueller01/garage-pio/blob/main/platformio.ini
I must be doing something wrong, it is still 404 for me :shrug:
Oh sorry. I added additional access for you. Hope it works now.
In the meantime I tried SDK 2.x and code crashed even earlier. Maybe I have to remove more code to narrow down the issue. It is really strange that some projects of mine work and others don't. Even that the code is almost the same now.
connected with xxx, channel 1
dhcp client start...
wifi evt: 0
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
wifi evt: 3
Station connected: 192.168.x.x
[WIFIM] Connected:
[WIFIM] SSID: xxx
[WIFIM] IP: 192.168.x.x
[WIFIM] MAC: 48:5D:35:7D:74:41
[WIFIM] CH: 1
[WIFIM] RSSI: -66
[WML] WiFi connected after num retries: 0
[WML] SSID=xxx,RSSI=-66
[WML] Channel=1,IP=192.168.x.x
[WML] bg: WiFi OK.
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Tue Jan 23 19:22:25 2024
mqttSetup: MQTT client id: ESP-0015AADA
mqttSetup: HomA system id: 123456-garage
mqttSetup: Attempting connection to server y:8883
[hostByName] request IP for: y
No poison after block at: 0x3fff2ef8, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Abort called
sp: 3ffffe60 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
ctx: cont
0x4020e4ec: make_unique<(anonymous namespace)::_dns_found_result, (anonymous namespace)::_dns_found_result> at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/unique_ptr.h:962
(inlined by) hostByNameImpl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:625
0x401006b0: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:246
0x4021e557: operator new(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/abi.cpp:41
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020e4ec: make_unique<(anonymous namespace)::_dns_found_result, (anonymous namespace)::_dns_found_result> at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/unique_ptr.h:962
(inlined by) hostByNameImpl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:625
0x4022a0fc: base64_encode_value at ??:?
0x40211ddc: BearSSL::WiFiClientSecureCtx::setFingerprint(char const*) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:659
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020e7dc: ESP8266WiFiGenericClass::hostByName(char const*, IPAddress&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:685
0x402129c5: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:220
0x4022a0fc: base64_encode_value at ??:?
0x4027f420: etharp_output at ??:?
0x3ffef010: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:89
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40207f05: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:423
0x4022841c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4020664c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x40209450: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:927
0x4021e987: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:260
0x40101701: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/cont.S:81
(and of course I erased flash before downgrading of the SDK ...)
Watch out for unchecked (s)printf
diff --git a/src/garage-arduino.cpp b/src/garage-arduino.cpp
index 85602ae..3088409 100644
--- a/src/garage-arduino.cpp
+++ b/src/garage-arduino.cpp
@@ -419,7 +420,7 @@ void mqttSetup() {
INFOF_P("%s: MQTT client id: %s" LF, __func__, host_name);
SAFE_FREE(m_homa_id);
- m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID) + 1));
+ m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID)) + 1);
sprintf_P(m_homa_id, PSTR(HOMA_SYSTEM_ID)); //, ESP.getChipId());
INFOF_P("%s: HomA system id: %s" LF, __func__, m_homa_id);
strlen_P
measures -1 byte, then sprintf_P
breaks the memory block by putting last char of the string and \0
into the poison block
For example, HOMA_SYSTEM_ID
is 123
. umm fails here for me as well when allocating tcp connection memory after noticing that neighbouring memory is broken
No poison after block at: 0x3fff2ba6, actual data: 0x33 0x0 0xa5 0xa5
0xa5
is the 'poison' byte which surrounds the allocation block in groups of four, but it is overwritten with 0x33
(char 3
) and 0x0
(aka null terminator), b/c allocation size is 2 and not the expected 4.
(gdb) x/16xb 0x3fff2ba6 - 8
0x3fff2b9e: 0x00 0x00 0xa5 0xa5 0xa5 0xa5 0x31 0x32
0x3fff2ba6: 0x33 0x00 0xa5 0xa5 0x00 0x00 0x00 0x00
Looks like 123\0
above. Pointing at the symbol specifically, it is definitely the neighbour and the reason of failure.
(gdb) p m_homa_id
$1 = 0x3fff2ba4 "123"
(gdb) x/16xb (uint8_t*)m_homa_id - 4
0x3fff2ba0: 0xa5 0xa5 0xa5 0xa5 0x31 0x32 0x33 0x00
0x3fff2ba8: 0xa5 0xa5 0x00 0x00 0x00 0x00 0x00 0x00
@mcspr you are incredible! Thank you sooo much! What a stupid mistake of mine. Shame on me. I owe you something. From my other project I changed
#define _HOMA_SYSTEM_ID "%06x-rhts" // numerical part (%06x) gets added during runtime
m_homa_id = (char *)malloc(8 + strlen_P(PSTR(_HOMA_SYSTEM_ID)));
sprintf_P(m_homa_id, PSTR(_HOMA_SYSTEM_ID), ESP.getChipId());
to
#define HOMA_SYSTEM_ID "123456-garage"
m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID) + 1));
sprintf_P(m_homa_id, PSTR(HOMA_SYSTEM_ID)); //, ESP.getChipId());
What a bad idea to write the +1 at the end at the slightly wrong position.
But good to know how to interpret the 'poison' bytes. Is there a documentation somewhere (tried to find something without success)? And good to know how to use the debugger here. Thanks. Thanks. Thanks.
A last question is still there. Is it possible to set another reset reason? This 'panic' resets the ESP and after reboot
ESP.getResetReason()
outputs Software/System restart
(REASON_SOFT_RESTART
) I would expect an Exception
and not like a 'normal' restart ...
But good to know how to interpret the 'poison' bytes. Is there a documentation somewhere (tried to find something without success)?
Only through reading the umm code and its comments. I think the example above could be a good addition to the gdb (https://github.com/esp8266/Arduino/blob/master/doc/gdb.rst) or faq (https://github.com/esp8266/Arduino/blob/master/doc/faq/a02-my-esp-crashes.rst , where it would corrupt some memory on purpose with an expected result
A last question is still there. Is it possible to set another reset reason? This 'panic' resets the ESP and after reboot ESP.getResetReason() outputs Software/System restart (REASON_SOFT_RESTART) I would expect an Exception and not like a 'normal' restart ...
Something overlooked. iirc, reset reason is passed through the 'custom callback' correctly but is not preserved during reboot as we read from SDK and SDK does not know of this special case
'Exception' as the term used here would mean Xtensa hw 'General Exception Causes' https://github.com/esp8266/Arduino/blob/de1029ffe0f85b40465ecb64218ef7ab3643ffa7/doc/exception_causes.rst
Value is also not preserved during the reboot correctly for this case, as it is only for display and not really an 'Exception' at all
Basic Infos
Platform
Settings in IDE
Problem Description
Software crashes with an User exception after connecting to WiFi and MQTT TLS
But it is not always at the same point in software. Sometimes it also starts the HTTP server and transmits several packages before the crash, like this:
I have no idea where to search for the problem. Can someone give me a hint (looking at the dump)? There seems to be enough heap available and fragmentation changes between 3 and 19 in the example above but does not constantly grow.
Another this that's strange to me is that
ESP.getResetReason()
outputsSoftware/System restart
after that. For me this looks like an Exception and not like a "normal" Software/System restart ...MCVE Sketch
Will follow, if needed.
Debug Messages