Closed mathieucarbou closed 3 weeks ago
I suspect this is caused by: CONFIG_LWIP_CHECK_THREAD_SAFETY
being activated ? See:
As I understand, CONFIG_LWIP_CHECK_THREAD_SAFETY
will fail if we do not properly lock ?
That is not a bad check per see, but right now, activating it could break a lof of downstream projects, but also Arduino components right ? Example of the stack trace above, which lies in the sntp feature.
Example:
This issue seems related: https://github.com/espressif/arduino-esp32/pull/10415
It is Matter that enabled it..
~/Desktop/ESP32/ESP-IDF-5/esp32-arduino-lib-builder/managed_components/espressif__esp_matter/connectedhomeip/connectedhomeip/config/esp32/components/chip/Kconfig:
220 bool "Enable LwIP Thread safety options"
221 default y
222: select LWIP_TCPIP_CORE_LOCKING
223 select LWIP_CHECK_THREAD_SAFETY
224 help
225 CHIP SDK performs LwIP core locking before calling an LwIP API.
226: To make the calls thread safe we have to enable LWIP_TCPIP_CORE_LOCKING.
227 Here, we are also enabling LWIP_CHECK_THREAD_SAFETY which will assert when
228 LwIP code gets called from any other context or without holding the LwIP lock.
AsyncUDP in this repo is already made to work with the locks. The fastest way is to have a look and see if you can solve it yourself. Not everything in Async is executed in the LwIP thread, so some locking will now be required. We can not get rid of Matter :)
But for the last stack trace, only sntp is involved... I am not using AsyncTCP in the second app, but AsyncTcpSock (bsd sockets). So the bug could come from where ?
For the first stack trace, do you mean that I should port in AsyncTCP these locks like they are done in AsyncUDP ?
That's impressive the amount of libs, web servers and mqtt clients will break when they will adopt 3.1.0.
I wish AsyncTCP were included in Arduino Core !
I wish AsyncTCP were included in Arduino Core !
I do plan to do this. Seems like it will be rather sooner than later š
I've placed some locks in the AsyncTCP lib, but still have some issues with sntp:
[ 5494][E][AsyncTCP.cpp:1492] begin(): [AsyncTCP] LOCK_TCPIP_CORE @ .pio/libdeps/pro-esp32/AsyncTCP/src/AsyncTCP.cpp:1492
[ 5515][E][AsyncTCP.cpp:1494] begin(): [AsyncTCP] UNLOCK_TCPIP_CORE @ .pio/libdeps/pro-esp32/AsyncTCP/src/AsyncTCP.cpp:1494
6m[ 5386][D][YASOLR][1][y-core] No electricity detected by ZCD module
[ 5534][E][AsyncTCP.cpp:1526] begin(): [AsyncTCP] LOCK_TCPIP_CORE @ .pio/libdeps/pro-esp32/AsyncTCP/src/AsyncTCP.cpp:1526
[ 5546][E][AsyncTCP.cpp:1529] begin(): [AsyncTCP] UNLOCK_TCPIP_CORE @ .pio/libdeps/pro-esp32/AsyncTCP/src/AsyncTCP.cpp:1529
[ 5428][I][YASOLR][1][y-pio] Enable NTP
assert failed: sntp_setoperatingmode /IDF/components/lwip/lwip/src/apps/sntp/sntp.c:728 (Required to lock TCPIP core functionality!)
Backtrace: 0x400835fd:0x3ffd7760 0x4008f09d:0x3ffd7780 0x4009565e:0x3ffd77a0 0x4012acd0:0x3ffd78d0 0x40116556:0x3ffd78f0 0x4010276f:0x3ffd7960 0x400ef37e:0x3ffd79c0 0x400ef445:0x3ffd7a00 0x4010586a:0x3ffd7a20 0x40105b2c:0x3ffd7a50 0x40105b6d:0x3ffd7a80 0x40105b91:0x3ffd7aa0 0x4008fba6:0x3ffd7ac0
#0 0x400835fd in panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:463
#1 0x4008f09d in esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c:92
#2 0x4009565e in __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:80
#3 0x4012acd0 in sntp_setoperatingmode at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/apps/sntp/sntp.c:729 (discriminator 1)
#4 0x40116556 in configTime at /Users/mat/.platformio/packages/framework-arduinoespressif32@src-e589a26bbeefbfd0fcf50e49e574e2c6/cores/esp32/esp32-hal-time.c:53
#5 0x4010276f in Mycila::NTPClass::sync(char const*, unsigned char) at .pio/libdeps/pro-esp32/MycilaNTP/src/MycilaNTP.cpp:66 (discriminator 1)
Ok so I've fixed AsyncTCP in https://github.com/mathieucarbou/AsyncTCP/pull/28:
Tested :
The only issue now remains sntp, and I have no control over that.
Is it a bug in Arduino Core ?
configTime at /Users/mat/.platformio/packages/framework-arduinoespressif32@src-e589a26bbeefbfd0fcf50e49e574e2c6/cores/esp32/esp32-hal-time.c:53
assert failed: sntp_setoperatingmode /IDF/components/lwip/lwip/src/apps/sntp/sntp.c:728 (Required to lock TCPIP core functionality!)
Backtrace: 0x400835fd:0x3ffd7760 0x4008f09d:0x3ffd7780 0x4009565e:0x3ffd77a0 0x4012acd0:0x3ffd78d0 0x40116556:0x3ffd78f0 0x4010276f:0x3ffd7960 0x400ef37e:0x3ffd79c0 0x400ef445:0x3ffd7a00 0x4010586a:0x3ffd7a20 0x40105b2c:0x3ffd7a50 0x40105b6d:0x3ffd7a80 0x40105b91:0x3ffd7aa0 0x4008fba6:0x3ffd7ac0
#0 0x400835fd in panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:463
#1 0x4008f09d in esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c:92
#2 0x4009565e in __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:80
#3 0x4012acd0 in sntp_setoperatingmode at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/apps/sntp/sntp.c:729 (discriminator 1)
#4 0x40116556 in configTime at /Users/mat/.platformio/packages/framework-arduinoespressif32@src-e589a26bbeefbfd0fcf50e49e574e2c6/cores/esp32/esp32-hal-time.c:53
#5 0x4010276f in Mycila::NTPClass::sync(char const*, unsigned char) at .pio/libdeps/pro-esp32/MycilaNTP/src/MycilaNTP.cpp:66 (discriminator 1)
From my point of view the sntp bug is a Arduino Core bug.
As a workaround I modified my sources in the following way:
#ifdef CONFIG_LWIP_TCPIP_CORE_LOCKING
if (!sys_thread_tcpip(LWIP_CORE_LOCK_QUERY_HOLDER))
LOCK_TCPIP_CORE();
#endif
configTzTime(m_timeZone.c_str(), m_ntpServer.c_str());
#ifdef CONFIG_LWIP_TCPIP_CORE_LOCKING
if (sys_thread_tcpip(LWIP_CORE_LOCK_QUERY_HOLDER))
UNLOCK_TCPIP_CORE();
#endif
My sketch is running fine now with Arduino Core 3.1.0-rc2, the mentioned workaround and the fix in AsyncTCP...
I have applied this patch in https://github.com/mathieucarbou/MycilaNTP and I confirm it works too.
can you please post a patch to the AsyncTCP repo and the ntp one to this repo? I will merge them asap
For NTP, yes, will do later.
For AsyncTCP, it is done: https://github.com/mathieucarbou/AsyncTCP.
I don't see the point in sending PRs to the original repo, except if there is a clear motivation to "resurrect" it and keep it alive and maintained, with a few collaborators added in the project, who can also deploy to registries (Arduino and Pio).
FYI the diff with original repo
There has been many fixes / improvements from ESPHome, community and me. At any point these can be merge back to the original repo.
I caught this one while chasing a system crash for a different issue. This is with Mathieu's updated AsyncTCP. The scenario is a simple EspAsyncWebServer (Mathieu's too) webserver/websockets with page data deliver via websockets. And in this case, an iPhone opened webpage (via Chrome) the page has stats delivered via websocket, but is not a page designed to be continually refreshed. The ws.cleanupClients() is being called and the following occured when the iPhone finally hits the auto-lock timeout. No additional investigation has been performed.
assert failed: tcp_arg /IDF/components/lwip/lwip/src/core/tcp.c:2031 (Required to lock TCPIP core functionality!)
0x40376932: panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c line 463
0x4037f0d9: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c line 92
0x4037f0d9: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c line 92
0x40385ccd: __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c line 80
0x40385ccd: __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c line 80
0x42061555: tcp_arg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 2031
0x42061555: tcp_arg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 2031
0x4203094d: AsyncClient::_error(signed char) at c:\Users\TerryPhillips\Documents\Arduino\libraries\AsyncTCP\src\AsyncTCP.cpp line 954
0x42030d33: _async_service_task(void*) at c:\Users\TerryPhillips\Documents\Arduino\libraries\AsyncTCP\src\AsyncTCP.cpp line 1414
0x4037fe52: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c
@microfoundry : good catch! Indeed I forgot this one: https://github.com/mathieucarbou/AsyncTCP/blob/main/src/AsyncTCP.cpp#L954 I will fix tomorrow morning and issue a new release.
@me-no-dev @NicoWallmeier : fyi PR opened for the sntp fix
@microfoundry : I have released AsyncTCP v3.2.12. Please let me know if you encounter other issues. It is still possible that some other code path need locking but I cannot find them right now.
on my code I got this error
assert failed: sntp_stop /IDF/components/lwip/lwip/src/apps/sntp/sntp.c:696 (Required to lock TCPIP core functionality!)
I used these libraries
on my code I got this error
assert failed: sntp_stop /IDF/components/lwip/lwip/src/apps/sntp/sntp.c:696 (Required to lock TCPIP core functionality!)
I used these libraries
include "time.h" #include "esp_sntp.h"
If you are not using the Arduino API you need to lock yourself the calls to sntp_* like I did: https://github.com/espressif/arduino-esp32/pull/10529/files
@microfoundry : I have released AsyncTCP v3.2.12. Please let me know if you encounter other issues. It is still possible that some other code path need locking but I cannot find them right now.
Thank you @mathieucarbou - that resolved the issue
@mathieucarbou it's up to you wether you will send a PR to the original repo. It's up to me wether will be "resurrected", but your ways do leave me wandering many things...
but your ways do leave me wandering many things...
If something is bothering you, you can tell me ;-) If something is bothering me, I will tell you (and I did, like above). Transparency is key, this is OSS, and we all work for the main benefits of others. So if you see something wrong to improve, please explain :-)
I just said I cannot afford to spend my little spare time sending a PR to fix all the projects I am not using, cannot use or cannot maintain, which is totally understandable. I would be glad to merge all the changes of my fork back to the original AsyncTCP repo though, but for me to spend time here, I don't want it to be for nothing and I would do it if there's a plan to add collaborators to the project, CI, deployment and also time allocated for maintenance, review, pr merge, etc. Things that I am able to do in my fork for now.
I would be glad to merge all the changes of my fork back to the original AsyncTCP repo
That would be great.
if there's a plan to add collaborators to the project, CI, deployment and also time allocated for maintenance, review, pr merge, etc.
Plan is to add it directly to the core and be fully supported here. I have not decided on the WebServer yet, but the TCP should be here, just like UDP. I do have some fun plans for it's future also.
Now I'm getting off-topic, but I can't resist:
I have not decided on the WebServer yet, but the TCP should be here, just like UDP. I do have some fun plans for it's future also.
Please also consider AsyncWebServer for this! It would be great to have an updated and maintained version of this masterpiece!
I would be glad to merge all the changes of my fork back to the original AsyncTCP repo
That would be great.
if there's a plan to add collaborators to the project, CI, deployment and also time allocated for maintenance, review, pr merge, etc.
Plan is to add it directly to the core and be fully supported here. I have not decided on the WebServer yet, but the TCP should be here, just like UDP. I do have some fun plans for it's future also.
Ok! I will put that on my list and will do it then. Fist need to finish other things but I will for sure have time to do it before the end of the year.
Going to kick this one again with:
assert failed: tcp_close_ext /IDF/components/lwip/lwip/src/core/tcp.c:492 (Required to lock TCPIP core functionality!)
0x4037f0d9: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c line 92
0x40385ccd: __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c line 80
0x42065aa6: tcp_close_ext at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 492
0x42065ad5: tcp_close at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 486
0x42033b40: AsyncServer::end() at c:\Users\TerryPhillips\Documents\Arduino\libraries\AsyncTCP\src\AsyncTCP.cpp line 1535
0x4203b4c1: AsyncWebServer::end() at c:\Users\TerryPhillips\Documents\Arduino\libraries\ESP_Async_WebServer\src\WebServer.cpp line 124
Going to kick this one again with:
assert failed: tcp_close_ext /IDF/components/lwip/lwip/src/core/tcp.c:492 (Required to lock TCPIP core functionality!) 0x4037f0d9: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/esp_system_chip.c line 92 0x40385ccd: __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c line 80 0x42065aa6: tcp_close_ext at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 492 0x42065ad5: tcp_close at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c line 486 0x42033b40: AsyncServer::end() at c:\Users\TerryPhillips\Documents\Arduino\libraries\AsyncTCP\src\AsyncTCP.cpp line 1535 0x4203b4c1: AsyncWebServer::end() at c:\Users\TerryPhillips\Documents\Arduino\libraries\ESP_Async_WebServer\src\WebServer.cpp line 124
I've mistakenly copy pasted and set the unlock at the wrong place š Fixing...
edit
=> v3.2.14
Description
I have several apps that I keep updated to lats Arduino RC.
3.1.0-RC1 work fine.
Since RC2 (3.1.0-RC2), I am observing several crashes with always the same error, in STA mode (not in AP mode).
Another app:
Sketch
Debug Message
Other Steps to Reproduce
No response
I have checked existing issues, online documentation and the Troubleshooting Guide