ESP32 crashes while using Events

[ismal]

I can observe crashes with my code on ESP32 on a very irregular basis and I was wondering is I am doing something wrong with ESPAsyncWebServer. My code is quite long but I think the conceptual part matters more. I have a webserver on ESP32 what serves a couple of webpages. Those webpages listen to /events on ESP32 and plot some graph real time using Highcharts. The real-time data comes to the webpages via /events, which are sent from the loop() function inside the ESP32 c++ part. Inside the loop() some computations are done and then messages are send:

loop(){ // do some computations for 100-300 msec

events.send(msg, "fttick", millis()); }

Now, everything works nicely when I stay on the page in the browser and watch the data. As soon as I start jumping between other webpages(served by ESP32), in one out of 5-10 jumps I catch a crash (with the error dump below). The same happens when I just start refreshing the same page in the browser, every 1-2 seconds (so after getting 4-5 events from the ESP32). It feels like 'client' pointer is disappearing while the message is supposed to be sent and the page is reloading.... but what is the proper way to do that (to handle closing and opening webpages on ESP32 while /events are sent)? The error is not easily reproducible, and it looks like the problem is on the ESPAsyncWebServer side, with all that async behavior. Would switching to websockets be a better option? (even though I do not need the two way communication).

The stacktrace is most of the time the same:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled. Core 1 register dump: PC : 0x401b0417 PS : 0x00060030 A0 : 0x800df59f A1 : 0x3ffb1c90 A2 : 0xbaad5678 A3 : 0x3f4059e3 A4 : 0x00000034 A5 : 0x00000001 A6 : 0x3ffd7478 A7 : 0x3ffb1c4c A8 : 0x800f4a28 A9 : 0x3ffb1c70 A10 : 0x00000002 A11 : 0x3f405d03 A12 : 0x80088db0 A13 : 0x3ffd64e0 A14 : 0x00000000 A15 : 0x3ffb0060 SAR : 0x0000000a EXCCAUSE: 0x0000001c EXCVADDR: 0xbaad5678 LBEG : 0x400014fd LEND : 0x4000150d LCOUNT : 0xffffffff

Backtrace: 0x401b0417:0x3ffb1c90 0x400df59c:0x3ffb1cb0 0x400dfcea:0x3ffb1cd0 0x400dfd45:0x3ffb1cf0 0x400dfd67:0x3ffb1d10 0x400dfdbd:0x3ffb1d30 0x400dbf49:0x3ffb1d60 0x400f6d85:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0

PC: 0x401b0417: AsyncClient::space() at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 749 EXCVADDR: 0xbaad5678

Decoding stack results 0x401b0417: AsyncClient::space() at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 749 0x400df59c: AsyncEventSourceMessage::send(AsyncClient*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 141 0x400dfcea: AsyncEventSourceClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 243 0x400dfd45: AsyncEventSourceClient::_queueMessage(AsyncEventSourceMessage*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 191 0x400dfd67: AsyncEventSourceClient::write(char const*, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 226 0x400dfdbd: AsyncEventSource::send(char const*, char const*, unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 318 0x400dbf49: loop() at src/main.cpp line 3691 0x400f6d85: loopTask(void*) at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 19 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

or it can also start without complaining about 'space()', something like this:

0x400dfcea: AsyncEventSourceClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 243 0x400dfd45: AsyncEventSourceClient::_queueMessage(AsyncEventSourceMessage*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 191 0x400dfd67: AsyncEventSourceClient::write(char const*, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 226 0x400dfdbd: AsyncEventSource::send(char const*, char const*, unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncEventSource.cpp line 318 0x400dbf49: loop() at src/main.cpp line 3691 0x400f6d85: loopTask(void*) at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 19 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

[ismal]

update: replacing events.send() in that loop() with websockes 'globalClient->text(...)' leads to much less troubles and crashes are happening (randomly) after tens of webpage refreshes with the following dump:

CORRUPT HEAP: Bad head at 0x3ffd8b28. Expected 0xabba1234 got 0x00000000 assertion "head != NULL" failed: file "/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c", line 214, function: multi_heap_free abort() was called at PC 0x4012c95f on core 1

Backtrace: 0x4008c454:0x3ffd6470 0x4008c685:0x3ffd6490 0x4012c95f:0x3ffd64b0 0x4008c0c9:0x3ffd64e0 0x40084b12:0x3ffd6500 0x40084f19:0x3ffd6520 0x4000bec7:0x3ffd6540 0x400f4fbb:0x3ffd6560 0x400f4fc9:0x3ffd6580 0x400e425f:0x3ffd65a0 0x400e06b7:0x3ffd65c0 0x400e0725:0x3ffd6600 0x401acba6:0x3ffd6620 0x400e1779:0x3ffd6640 0x4017e0fb:0x3ffd6660 0x4017e264:0x3ffd6690 0x4017e8b5:0x3ffd66b0 0x40088b9d:0x3ffd66e0

Decoding stack results 0x4008c454: invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 155 0x4008c685: abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 170 0x4012c95f: __assert_func at ../../../.././newlib/libc/stdlib/assert.c line 63 0x4008c0c9: multi_heap_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c line 214 0x40084b12: heap_caps_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c line 268 0x40084f19: _free_r at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c line 42 0x400f4fbb: String::invalidate() at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/WString.cpp line 140 0x400f4fc9: String::~String() at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/WString.cpp line 124 0x400e425f: AsyncWebServerRequest::~AsyncWebServerRequest() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/WebRequest.cpp line 80 0x400e06b7: AsyncWebSocketClient::AsyncWebSocketClient(AsyncWebServerRequest*, AsyncWebSocket*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 496 0x400e0725: AsyncWebSocketResponse::_ack(AsyncWebServerRequest*, unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 1296 0x401acba6: AsyncWebServerRequest::_onAck(unsigned int, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/WebRequest.cpp line 201 0x400e1779: std::_Function_handler >::_M_invoke(const std::_Any_data &, Users/stealth/Downloads/firmware.elf, CU 0x1040fc, DIE 0x11cb53>, Users/stealth/Downloads/firmware.elf, CU 0x1040fc, DIE 0x11cb58>, Users/stealth/Downloads/firmware.elf, CU 0x1040fc, DIE 0x11cb5d>, Users/stealth/Downloads/firmware.elf, CU 0x1040fc, DIE 0x11cb62>) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/WebRequest.cpp line 73 0x4017e0fb: AsyncClient::_sent(tcp_pcb*, unsigned short) at /Users/stealth/.platformio/packages/toolchain-xtensa32/xtensa-esp32-elf/include/c++/5.2.0/functional line 2271 0x4017e264: AsyncClient::_s_sent(void*, tcp_pcb*, unsigned short) at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 1203 0x4017e8b5: _async_service_task(void*) at /Users/stealth/.platformio/lib/AsyncTCP_ID1826/src/AsyncTCP.cpp line 165 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

Sometimes, there is also an error even without a page refresh, but after 1000 of sent messages int he form:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled. Core 1 register dump: PC : 0x400e0f36 PS : 0x00060230 A0 : 0x800e0ff6 A1 : 0x3ffb1c10 A2 : 0x3ffd75d8 A3 : 0x00000000 A4 : 0x3ffd7600 A5 : 0x3ffd74b0 A6 : 0x00000038 A7 : 0x2c382c34 A8 : 0x800e0ec0 A9 : 0x3ffb1bf0 A10 : 0x56783981 A11 : 0x40084f08 A12 : 0x3ffc21e8 A13 : 0x0e1d457c A14 : 0x00000000 A15 : 0x00000001 SAR : 0x00000010 EXCCAUSE: 0x0000001c EXCVADDR: 0x00000014 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0x00000000

Backtrace: 0x400e0f36:0x3ffb1c10 0x400e0ff3:0x3ffb1c30 0x400e1017:0x3ffb1c50 0x400e105f:0x3ffb1c70 0x400d4a09:0x3ffb1c90 0x400d68d1:0x3ffb1d30 0x400dbcad:0x3ffb1d50 0x400f6d71:0x3ffb1fb0 0x40088b9d:0x3ffb1fd0

PC: 0x400e0f36: AsyncWebSocketClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 542 EXCVADDR: 0x00000014

Decoding stack results 0x400e0f36: AsyncWebSocketClient::_runQueue() at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 542 0x400e0ff3: AsyncWebSocketClient::_queueMessage(AsyncWebSocketMessage*) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 566 0x400e1017: AsyncWebSocketClient::text(char const*, unsigned int) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 772 0x400e105f: AsyncWebSocketClient::text(String const&) at /Users/stealth/.platformio/lib/ESP Async WebServer_ID306/src/AsyncWebSocket.cpp line 784 0x400d4a09: sendTXinfoViaWebsocks() at src/main.cpp line 370 0x400d68d1: doOneMeasurement(bool) at src/main.cpp line 1038 0x400dbcad: loop() at src/main.cpp line 3626 0x400f6d71: loopTask(void*) at /Users/stealth/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 19 0x40088b9d: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

[zekageri]

same

[tonydee2377]

same. It would seem a problem on the web part, I read that the problem should be in AsyncEventSource and AsyncTCP. We should wait for a more stable release of these libraries. Anyway in my case I solved it, because except for two events being updated every 1s, I have many other event.send that I send only when I request it by pressing a button.

Waiting for a solution

[Gamelauncher]

I have the issue here as well ending in a: CORRUPT HEAP: Bad head at 0x3ffd8b28. Expected 0xabba1234 got 0x00000000 assertion

I have a polling setup every 5 seconds. Crash happens often when the polling coincides with another request.

[stevesch]

Any leads on this? I'm using AsyncEventSource and am hitting these crashes in _runQueue. There appears to be plenty of memory. I'll continue to investigate if there is some property of particular events being sent, but I think the params are all reasonable and valid.

[stevesch]

Incidents in may case seem to increase greatly when more clients connect, i.e. connect to the ESP32 using several tabs in the browser and it crashes quickly. Memory usage increases a bit, but nothing that would seem fatal-- heap free ~183K, quickly reloading the page served by the ESP32 a few times usually results in a crash and restart (heap min free observed to drop to ~113K, page is sending several server-side events, crash is in AsyncEventSource _runQueue as above). Is there perhaps a chance of the client list being changed while being iterated by something? (causing the iteration to be invalid, maybe causing crashes later)

[Gamelauncher]

I've managed to make it stable.

Using: AsyncTCP ( patched up version ) AsyncMqtt ( dev version ) ESPAsyncWebserver ( patched up version )

I've downgraded the ESP32 Arduino from 1.0.5 ( 1.0.6 has the same problems for me ) to 1.0.4 and all the problems went away.

[stevesch]

@Gamelauncher great, and thanks for the info! I'm using PlatformIO, but it sounds like a bug in the latest framework, or an incompatibility between the latest framework and one of the libs. I'll see if I can get some combination of these lib and framework versions working for me in the PlatformIO environment.

[zekageri]

I've managed to make it stable.

Using: AsyncTCP ( patched up version ) AsyncMqtt ( dev version ) ESPAsyncWebserver ( patched up version )

I've downgraded the ESP32 Arduino from 1.0.5 ( 1.0.6 has the same problems for me ) to 1.0.4 and all the problems went away.

Where is the patched AsyncTCP lib?

[Gamelauncher]

https://github.com/ul-gh/AsyncTCP/tree/Fix_%23111

Some of the forks contain multiple additional fixes. I believe I used this one.

[stevesch]

@Gamelauncher which branch/version for ESPAsyncWebserver did you use (the "patched up version")?

[Gamelauncher]

Try this one: https://github.com/ul-gh/ESPAsyncWebServer/tree/dev

There are allot of pull requests open some people have actually merged them in the forks.

[zekageri]

Thank you. I will give it a try!

[stevesch]

These branches helped a lot. I have yet to verify 100% fix, but it's looking good.

[stevesch]

What are the odds of getting these changes merged back into the origins? Is me-no-dev still maintaining these libs? They're great, btw, not to complain. It would be good to have these fixes in the official libraries available in the library manager and PIO (or at least the origin git repo URL).

[zekageri]

Unfortunately i think me_no_dev has a lot going on with new esps integration and such :( The fix is crashed for me btw. My esp is port forwarded and if more than 3 clients wants to connect from outside the network, the esp crashes. I dont know if it is my fault on the routes or not, yet.

[tonydee2377]

I confirm the library fixes. Even returning everything to the origin with a refresh page of 1s (EVENT_INTERVAL_MS = 1000), I no longer crash. Well, I'll use these libraries as official. Thanks a lot

[stevesch]

For other PlatformIO users, these are the lib_deps I'm using for the patched branches (semicolon lines are commented out, replaced by ul-gh repo libs:

lib_deps = ; me-no-dev/AsyncTCP@^1.1.1 ; Specific branch for AsyncTCP fixes: https://github.com/ul-gh/AsyncTCP.git#ffa499e53175942f2d7c00e72c939524fda0053e

; me-no-dev/ESP Async WebServer@^1.2.3 ; Specific branch for ESPAsyncWebServer fixes: https://github.com/ul-gh/ESPAsyncWebServer.git#3b8c659cf83da49ed95a96a85024a69237ad9711

If this is for a pre-existing project, you'll have to delete the lib_deps folder in the project so PlatformIO will download the patched libs.

[zekageri]

I confirm the library fixes. Even returning everything to the origin with a refresh page of 1s (EVENT_INTERVAL_MS = 1000), I no longer crash. Well, I'll use these libraries as official. Thanks a lot

What do you mean by returning to the origin? To the original me_no_dev libraries? How is it possible if no fix was merged?

[zekageri]

For other PlatformIO users, these are the lib_deps I'm using for the patched branches (semicolon lines are commented out, replaced by ul-gh repo libs:

lib_deps = ; me-no-dev/AsyncTCP@^1.1.1 ; Specific branch for AsyncTCP fixes: https://github.com/ul-gh/AsyncTCP.git#ffa499e53175942f2d7c00e72c939524fda0053e

; me-no-dev/ESP Async WebServer@^1.2.3 ; Specific branch for ESPAsyncWebServer fixes: https://github.com/ul-gh/ESPAsyncWebServer.git#3b8c659cf83da49ed95a96a85024a69237ad9711

If this is for a pre-existing project, you'll have to delete the lib_deps folder in the project so PlatformIO will download the patched libs.

Thank you! I will give this a try too.

[tonydee2377]

No, I intended to go back to my old unedited sketch, so I had problems before and now my old project has no more problems. the original sketch not libraries.

[zekageri]

No, I intended to go back to my old unedited sketch, so I had problems before and now my old project has no more problems. the original sketch not libraries.

Oh i see. I'am testing these two libs now:

AsyncTCP by ul-gh ESPAsyncWebServer by ul-gh

I'll let it run for a couple of days and we will see. I'am using a lot of things along with that. Ntp time fetch, Weather data fetch, spi screen, i2c rtc module, heavy serial activities with modbus protocol and things like that. Http things going with a LAN8720 eth chip meanwhile.

[zekageri]

Still crashing but with other backtrace. :(

CORRUPT HEAP: Bad tail at 0x3ffd8a70. Expected 0xbaad5678 got 0xbaad5600
CORRUPT HEAP: multi_heap.c:432 detected at 0x3ffd8a74
abort() was called at PC 0x400930f3 on core 1

ELF file SHA256: 0000000000000000

Backtrace: 0x4008eb28:0x3ffd78e0 0x4008eda1:0x3ffd7900 0x400930f3:0x3ffd7920 
0x400937b4:0x3ffd7940 0x400827e9:0x3ffd7960 0x4008281a:0x3ffd7980 0x4008999d:0x3ffd79a0 0x4000beaf:0x3ffd79c0 0x4008f66e:0x3ffd79e0 0x4008f8bc:0x3ffd7a00 
0x40089a5e:0x3ffd7a20 0x40089a89:0x3ffd7a40 0x40089bdd:0x3ffd7a70 0x4012df7f:0x3ffd7a90 0x40129fc9:0x3ffd7d50 0x40129f49:0x3ffd7da0 0x40093801:0x3ffd7dd0 
0x40082886:0x3ffd7df0 0x400899a9:0x3ffd7e10 0x4000bec7:0x3ffd7e30 0x4019b879:0x3ffd7e50 0x400ece53:0x3ffd7e70 0x400ed611:0x3ffd7e90 0x400ed655:0x3ffd7ec0 
0x400e9735:0x3ffd7ee0 0x400e98cc:0x3ffd7f10 0x400ea015:0x3ffd7f30 0x4008fda6:0x3ffd7f60
  #0  0x4008eb28:0x3ffd78e0 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715 
  #1  0x4008eda1:0x3ffd7900 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:715        
  #2  0x400930f3:0x3ffd7920 in multi_heap_assert at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap.c:380
      (inlined by) multi_heap_malloc_impl at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap.c:432
  #3  0x400937b4:0x3ffd7940 in multi_heap_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:305
  #4  0x400827e9:0x3ffd7960 in heap_caps_malloc at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:354
  #5  0x4008281a:0x3ffd7980 in heap_caps_malloc_default at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:354
  #6  0x4008999d:0x3ffd79a0 in _malloc_r at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c:37 
  #7  0x4000beaf:0x3ffd79c0 in ?? ??:0
  #8  0x4008f66e:0x3ffd79e0 in xQueueGenericCreate at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:2038
  #9  0x4008f8bc:0x3ffd7a00 in xQueueCreateMutex at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:2038
  #10 0x40089a5e:0x3ffd7a20 in lock_init_generic at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:79
  #11 0x40089a89:0x3ffd7a40 in lock_acquire_generic at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:134
  #12 0x40089bdd:0x3ffd7a70 in _lock_acquire_recursive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:171
  #13 0x4012df7f:0x3ffd7a90 in _vfiprintf_r at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/vfprintf.c:860 (discriminator 2)
  #14 0x40129fc9:0x3ffd7d50 in fiprintf at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/fiprintf.c:50
  #15 0x40129f49:0x3ffd7da0 in __assert_func at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:59 (discriminator 8)
  #16 0x40093801:0x3ffd7dd0 in multi_heap_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/multi_heap_poisoning.c:305
  #17 0x40082886:0x3ffd7df0 in heap_caps_free at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/heap/heap_caps.c:354
  #18 0x400899a9:0x3ffd7e10 in _free_r at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/syscalls.c:42
  #19 0x4000bec7:0x3ffd7e30 in ?? ??:0
  #20 0x4019b879:0x3ffd7e50 in operator delete(void*) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/del_op.cc:46
  #21 0x400ece53:0x3ffd7e70 in LinkedList<AsyncWebSocketMessage*, LinkedListNode>::remove(AsyncWebSocketMessage* const&) at lib\ESPAsyncWebServer-master\src/StringArray.h:53
      (inlined by) AsyncWebSocketClient::_runQueue() at lib\ESPAsyncWebServer-master\src/AsyncWebSocket.cpp:539      
  #22 0x400ed611:0x3ffd7e90 in AsyncWebSocketClient::_onAck(unsigned int, unsigned int) at lib\ESPAsyncWebServer-master\src/StringArray.h:53
  #23 0x400ed655:0x3ffd7ec0 in std::_Function_handler<void (void*, AsyncClient*, unsigned int, unsigned int), AsyncWebSocketClient::AsyncWebSocketClient(AsyncWebServerRequest*, AsyncWebSocket*)::{lambda(void*, AsyncClient*, unsigned int, unsigned int)#4}>::_M_invoke(std::_Any_data const&, void*&&, AsyncClient*&&, unsigned int&&, AsyncClient*&&) at lib\ESPAsyncWebServer-master\src/StringArray.h:53
      (inlined by) _M_invoke at c:\users\pc\.platformio\packages\toolchain-xtensa32\xtensa-esp32-elf\include\c++\5.2.0/functional:1871
  #24 0x400e9735:0x3ffd7ee0 in std::function<void (void*, AsyncClient*, unsigned int, unsigned int)>::operator()(void*, AsyncClient*, unsigned int, unsigned int) const at lib\AsyncTCP-master\src/AsyncTCP.cpp:1065
      (inlined by) AsyncClient::_sent(tcp_pcb*, unsigned short) at lib\AsyncTCP-master\src/AsyncTCP.cpp:917
  #25 0x400e98cc:0x3ffd7f10 in AsyncClient::_s_sent(void*, tcp_pcb*, unsigned short) at lib\AsyncTCP-master\src/AsyncTCP.cpp:1065
  #26 0x400ea015:0x3ffd7f30 in _async_service_task(void*) at lib\AsyncTCP-master\src/AsyncTCP.cpp:1065
      (inlined by) _async_service_task at lib\AsyncTCP-master\src/AsyncTCP.cpp:197
  #27 0x4008fda6:0x3ffd7f60 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...

It is crashing when i want to send a websocket text message like this:

if(globalClient != NULL && globalClient->status() == WS_CONNECTED){
        if (ws.count() > 0){
          ws.textAll(msg); // msg is a String
        }
      }
 }

[stevesch]

@zekageri I'm not using websockets directly, just heavily using AsyncEventSource, and I've encountered no such issues, so it might be specifically in the websockets portion of the code. Don't rule out something completely independent though-- once my networking issues were resolved I was hitting another crash in some unrelated encoder code related to an ISR function specified without IRAM_ATTR. I got a similar backtrace as before and I immediately thought "ESPAsyncWebServer library was not fixed," but was [fortunately] wrong (it is stable for my limited use case at least).

[zekageri]

Yeah, it seems to me that my crashes was because of low heap for me. Iam not sure yet, but i moved some big json Strings from internal to external ram and it seems stable for a day now. With fixed libs ofc

[zekageri]

The problem still there. Iam sending a not that big JSON String every two seconds and eventually it crashes on ws send and ws free. :(

[rikki78]

Could it be, no client is connected anymore when it crashes? Closing the connection doesn't work always. As https://github.com/me-no-dev/ESPAsyncWebServer/issues/802

[hel-mut]

I have also crashes of an ESP32 with AsyncEventServer. The crash happens always at the AsyncEventSource.cpp at line 244: void AsyncEventSourceClient::_runQueue(){ while(!_messageQueue.isEmpty() && _messageQueue.front()->finished()){ _messageQueue.remove(_messageQueue.front()); }

for(auto i = _messageQueue.begin(); i != _messageQueue.end(); ++i) { if(!(i)->sent()) // this is line 244 (i)->send(_client); } }

I also noticed, that sometimes the same event is sent two times, if more than one event.sent command follows the ping command without any delay, eg: events.send("ping",NULL,millis()); events.send(String(T.out).c_str(),"temperature",millis()); events.send(String(RH.out).c_str(),"humidity",millis()); events.send(String(CO2.out).c_str(),"co2",millis()); events.send(String(TVOC.out).c_str(),"tvoc",millis()); events.send(String(pressure).c_str(),"pressure",millis());

In my opinion there must be accesses to the messageQueue from different tasks (of the 2 cores of ESP32), that are not locked. I think the crash happens, when an event is sent and the message is removed from the messageQueue, while the AsyncEventServer is in the for i loop. In this case the end value i != _messageQueue.end() is not updated in the loop and therefore an access to an removed message happens.

[stale[bot]]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.

[vazir]

I've managed to make it stable.

Using: AsyncTCP ( patched up version ) AsyncMqtt ( dev version ) ESPAsyncWebserver ( patched up version )

I've downgraded the ESP32 Arduino from 1.0.5 ( 1.0.6 has the same problems for me ) to 1.0.4 and all the problems went away.

I've got similar issue, when intensively switching on/off over MQTT (just a gpio switch), issue look very similar... did you find a cure finally? Tried the libs you mentioned - all the same...