"free() target pointer is outside heap areas" when using `std::stream_buf` (concurrently) (IDFGH-9590)

wuyuanyi135 commented 1 year ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.1-dev-3818-g7f60b8ec32

Operating System used.

Windows

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

PowerShell

Development Kit.

ESP32-S3-WROOM-1-N16R2

Power Supply used.

USB

What is the expected behavior?

I have a project that utilizes two streams (inherented from std::basic_streambuf<uint8_t>) to save RAM:

Data serialization --- stream 1 ---> SDCard
SDCard --- stream 2 ---> HTTP Chunked Response

The first stream was used periodically to write data to SDCard partitions. The second stream was used to dump the written data to a HTTP client. They all work fine when used independently.

What is the actual behavior?

When dumping data, I want to use the event loop to notify all other stream 1 to flush their cached data into SDCard. Therefore, the stream 1 and stream 2 may be constructed concurrently. (The access to SDCard is mutex-protected).

In this case, I got a crash when constructing the streams:

assert failed: heap_caps_free heap_caps.c:375 (heap != NULL && \"free() target pointer is outside heap areas\")

By backtracing the stack I found it was a error thrown when constructing std::locale. I found a similar issue https://github.com/espressif/esp-idf/issues/5150 but the code did not reproduce the error.

Steps to reproduce.

I could not share the whole project and it may need some work to get the MWE from my current project.

I will share the elf and the coredump here first. Let me know if more information is needed.

Debug Logs.

#0  0x40375e8d in panic_abort (details=0x3fcda2c0 "assert failed: heap_caps_free heap_caps.c:375 (heap != NULL && \"free() target pointer is outside heap areas\")")
    at C:/Espressif/frameworks/esp-idf-master/components/esp_system/panic.c:451
451         *((volatile int *) 0) = 0; // NOLINT(clang-analyzer-core.NullDereference) should be an invalid operation on targets
[Current thread is 1 (process 1070443412)]
(gdb)
(gdb) bt
#0  0x40375e8d in panic_abort (details=0x3fcda2c0 "assert failed: heap_caps_free heap_caps.c:375 (heap != NULL && \"free() target pointer is outside heap areas\")")
    at C:/Espressif/frameworks/esp-idf-master/components/esp_system/panic.c:451
#1  0x4037fcb4 in esp_system_abort (details=0x3fcda2c0 "assert failed: heap_caps_free heap_caps.c:375 (heap != NULL && \"free() target pointer is outside heap areas\")")
    at C:/Espressif/frameworks/esp-idf-master/components/esp_system/port/esp_system_chip.c:78
#2  0x40387130 in __assert_func (file=<optimized out>, line=<optimized out>, func=<optimized out>, expr=<optimized out>)
    at C:/Espressif/frameworks/esp-idf-master/components/newlib/assert.c:81
#3  0x40376818 in heap_caps_free (ptr=0x3fcaaae4 <(anonymous namespace)::facet_vec>) at C:/Espressif/frameworks/esp-idf-master/components/heap/heap_caps.c:375
#4  0x4038718c in free (ptr=0x3fcaaae4 <(anonymous namespace)::facet_vec>) at C:/Espressif/frameworks/esp-idf-master/components/newlib/heap.c:39
#5  0x420d2050 in operator delete (ptr=0x3fcaaae4 <(anonymous namespace)::facet_vec>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49
#6  0x420d2080 in operator delete[] (ptr=0x3fcaaae4 <(anonymous namespace)::facet_vec>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/libsupc++/del_opv.cc:35
#7  0x420d32a6 in std::locale::_Impl::_M_install_facet (this=0x3fcaabcc <(anonymous namespace)::c_locale_impl>, __idp=<optimized out>,
    __fp=0x3fcaa1fc <(anonymous namespace)::codecvt_c16>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale.cc:361
#8  0x420d3a50 in std::locale::_Impl::_M_init_facet<std::codecvt<char16_t, char, _mbstate_t> > (__facet=0x3fcaa1fc <(anonymous namespace)::codecvt_c16>,
    this=0x3fcaabcc <(anonymous namespace)::c_locale_impl>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/build/build-cc-gcc-final/xtensa-esp32s3-elf/no-rtti/libstdc++-v3/include/bits/locale_classes.h:612
#9  std::locale::_Impl::_Impl (this=0x3fcaabcc <(anonymous namespace)::c_locale_impl>, __refs=<optimized out>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale_init.cc:571
#10 0x420d3b70 in std::locale::_S_initialize_once ()
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale_init.cc:333
#11 std::locale::_S_initialize_once () at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale_init.cc:323
#12 0x42002ca0 in pthread_once (once_control=0x3fc9bdd0 <std::locale::_S_once>, init_routine=0x420d3b58 <std::locale::_S_initialize_once()>)
    at C:/Espressif/frameworks/esp-idf-master/components/pthread/pthread.c:503
#13 0x420d3b93 in __gthread_once (__once=<optimized out>, __func=0x420d3b58 <std::locale::_S_initialize_once()>)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/build/build-cc-gcc-final/xtensa-esp32s3-elf/no-rtti/libstdc++-v3/include/xtensa-esp32s3-elf/bits/gthr-default.h:700
#14 std::locale::_S_initialize () at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale_init.cc:343
#15 0x420d3bb1 in std::locale::locale (this=0x3fcda60c)
    at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32s3-elf/src/gcc/libstdc++-v3/src/c++98/locale_init.cc:271
#16 0x420225ad in HTTPResponseStream::HTTPResponseStream (req=0x3fcd7b14, this=0x3fcda5f0)
    at C:/Users/wuyua/Documents/Project/CrystalVisionLtd/Embedded/unilab/components/unilab/src/utils/http_utils.h:282
#17 data_handler (req=0x3fcd7b14) at C:/Users/wuyua/Documents/Project/CrystalVisionLtd/Embedded/unilab/components/unilab/src/service/http_server.cpp:151
#18 0x4207a94e in httpd_uri (hd=0x3fcd7aa0) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_uri.c:329
#19 0x420798bb in httpd_parse_req (hd=0x3fcd7aa0) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_parse.c:662
#20 httpd_req_new (hd=0x3fcd7aa0, sd=<optimized out>) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_parse.c:790
#21 0x42079fd0 in httpd_sess_process (hd=0x3fcd7aa0, session=0x3fcd7fd0) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_sess.c:423
#22 0x420788d5 in httpd_process_session (context=0x3fcda860, session=0x3fcd7fd0) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_main.c:247
#23 httpd_process_session (session=0x3fcd7fd0, context=0x3fcda860) at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_main.c:232
#24 0x421049c4 in httpd_sess_enum (hd=<optimized out>, enum_function=0x42078898 <httpd_process_session>, context=0x3fcda860)
    at C:/Espressif/frameworks/esp-idf-master/components/esp_http_server/src/httpd_sess.c:50

More Information.

ELF and coredump debug.zip

wuyuanyi135 commented 1 year ago

I don't have access to the source of stdc++ implmentation. I will post some possible related links here

https://www.mail-archive.com/gcc-bugs@gcc.gnu.org/msg617635.html

https://gcc.gnu.org/legacy-ml/gcc-help/2017-05/msg00164.html

0xjakob commented 1 year ago

@wuyuanyi135 Thanks, we will take a look.

BTW., you can find libstdc++ for toolchains dedicated for ESP chips here: https://github.com/espressif/gcc/tree/esp_based_on_11_2_0/libstdc%2B%2B-v3.

wuyuanyi135 commented 1 year ago

@0xjakob Thank you for the information. Let me know if more information is needed.

espressif / esp-idf