python / cpython

The Python programming language
https://www.python.org
Other
63.49k stars 30.41k forks source link

Possible `heap-use-after-free` in ctypes in Python 3.12 #113576

Open jiridanek opened 10 months ago

jiridanek commented 10 months ago

Crash report

What happened?

I have a binary which is embedding python. I want to test this binary with address sanitizer, and leak sanitizer. Regular python3.12 build contained in Fedora 39 causes lsan to report leaks even with PYTHONMALLOC=malloc_debug, which with previous versions gave me clean run.

Therefore, I tried python3-debug, which is a debug build of python packaged in Fedora. With that, I hit

and I was playing around a bit. When I hit the mentioned issue, I was using PYTHONMALLOC="malloc_debug". I wanted to try other values for this variable. With pymalloc_debug, I got the same assertion.

With malloc, I got

=================================================================
==1716614==ERROR: AddressSanitizer: heap-use-after-free on address 0x6040000aba78 at pc 0x7f2ba546eb32 bp 0x7f2b9e24e520 sp 0x7f2b9e24dce0
READ of size 3 at 0x6040000aba78 thread T4
    #0 0x7f2ba546eb31 in __interceptor_memcpy (/lib64/libasan.so.8+0x6eb31) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f2ba4cd6d8a in _copy_characters (/lib64/libpython3.12d.so.1.0+0x2d6d8a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #2 0x7f2ba4cd7831 in _PyUnicode_FastCopyCharacters (/lib64/libpython3.12d.so.1.0+0x2d7831) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #3 0x7f2ba4cf54ef in PyUnicode_Concat (/lib64/libpython3.12d.so.1.0+0x2f54ef) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #4 0x7f2ba0a44b51 in CreateSwappedType (/usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312d-x86_64-linux-gnu.so+0xcb51) (BuildId: cdb5a61ddd026358017a72a3d3405dd431e31b72)
    #5 0x7f2ba0a4552c in PyCSimpleType_new (/usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312d-x86_64-linux-gnu.so+0xd52c) (BuildId: cdb5a61ddd026358017a72a3d3405dd431e31b72)
    #6 0x7f2ba4c958c3 in type_call (/lib64/libpython3.12d.so.1.0+0x2958c3) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #7 0x7f2ba4bf8792 in _PyObject_MakeTpCall (/lib64/libpython3.12d.so.1.0+0x1f8792) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #8 0x7f2ba4bf825c in _PyObject_FastCallDictTstate (/lib64/libpython3.12d.so.1.0+0x1f825c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #9 0x7f2ba4bf8389 in PyObject_VectorcallDict (/lib64/libpython3.12d.so.1.0+0x1f8389) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #10 0x7f2ba4d5afd1 in builtin___build_class__ (/lib64/libpython3.12d.so.1.0+0x35afd1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #11 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #12 0x7f2ba4bf7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #13 0x7f2ba4bf8b98 in PyObject_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8b98) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #14 0x7f2ba4d7b662 in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x37b662) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #15 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #16 0x7f2ba4d86a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #17 0x7f2ba4d63933 in PyEval_EvalCode (/lib64/libpython3.12d.so.1.0+0x363933) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #18 0x7f2ba4d5c678 in builtin_exec_impl (/lib64/libpython3.12d.so.1.0+0x35c678) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #19 0x7f2ba4d59a3a in builtin_exec (/lib64/libpython3.12d.so.1.0+0x359a3a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #20 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #21 0x7f2ba4bf8909 in _PyVectorcall_Call (/lib64/libpython3.12d.so.1.0+0x1f8909) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #22 0x7f2ba4bf8cf1 in _PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8cf1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #23 0x7f2ba4bf8dcc in PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8dcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #24 0x7f2ba4d814dc in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x3814dc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #25 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #26 0x7f2ba4d86a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #27 0x7f2ba4bf8fdd in _PyFunction_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8fdd) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #28 0x7f2ba4bf7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #29 0x7f2ba4bfa545 in object_vacall (/lib64/libpython3.12d.so.1.0+0x1fa545) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #30 0x7f2ba4bfa84a in PyObject_CallMethodObjArgs (/lib64/libpython3.12d.so.1.0+0x1fa84a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #31 0x7f2ba4ddba6b in import_find_and_load (/lib64/libpython3.12d.so.1.0+0x3dba6b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #32 0x7f2ba4ddbdb8 in PyImport_ImportModuleLevelObject (/lib64/libpython3.12d.so.1.0+0x3dbdb8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #33 0x7f2ba4d88907 in import_name (/lib64/libpython3.12d.so.1.0+0x388907) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #34 0x7f2ba4d76e96 in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x376e96) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #35 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #36 0x7f2ba4d86a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #37 0x7f2ba4d63933 in PyEval_EvalCode (/lib64/libpython3.12d.so.1.0+0x363933) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #38 0x7f2ba4d5c678 in builtin_exec_impl (/lib64/libpython3.12d.so.1.0+0x35c678) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #39 0x7f2ba4d59a3a in builtin_exec (/lib64/libpython3.12d.so.1.0+0x359a3a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #40 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #41 0x7f2ba4bf8909 in _PyVectorcall_Call (/lib64/libpython3.12d.so.1.0+0x1f8909) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #42 0x7f2ba4bf8cf1 in _PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8cf1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #43 0x7f2ba4bf8dcc in PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8dcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #44 0x7f2ba4d814dc in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x3814dc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #45 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #46 0x7f2ba4d86a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #47 0x7f2ba4bf8fdd in _PyFunction_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8fdd) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #48 0x7f2ba4bf7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #49 0x7f2ba4bfa545 in object_vacall (/lib64/libpython3.12d.so.1.0+0x1fa545) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #50 0x7f2ba4bfa84a in PyObject_CallMethodObjArgs (/lib64/libpython3.12d.so.1.0+0x1fa84a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #51 0x7f2ba4ddba6b in import_find_and_load (/lib64/libpython3.12d.so.1.0+0x3dba6b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #52 0x7f2ba4ddbdb8 in PyImport_ImportModuleLevelObject (/lib64/libpython3.12d.so.1.0+0x3dbdb8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #53 0x7f2ba4d5b1ee in builtin___import___impl (/lib64/libpython3.12d.so.1.0+0x35b1ee) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #54 0x7f2ba4d591d8 in builtin___import__ (/lib64/libpython3.12d.so.1.0+0x3591d8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #55 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #56 0x7f2ba4bf7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #57 0x7f2ba4bf9595 in _PyObject_CallFunctionVa (/lib64/libpython3.12d.so.1.0+0x1f9595) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #58 0x7f2ba4bf96e5 in PyObject_CallFunction (/lib64/libpython3.12d.so.1.0+0x1f96e5) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #59 0x7f2ba4ddc3f5 in PyImport_Import (/lib64/libpython3.12d.so.1.0+0x3dc3f5) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #60 0x7f2ba4ddae85 in PyImport_ImportModule (/lib64/libpython3.12d.so.1.0+0x3dae85) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #61 0x5e5700 in qd_python_setup /home/jdanek/repos/skupper-router/src/python_embedded.c:841
    #62 0x57b5a0 in qd_python_initialize /home/jdanek/repos/skupper-router/src/python_embedded.c:58
    #63 0x57b5a0 in qd_dispatch /home/jdanek/repos/skupper-router/src/dispatch.c:137
    #64 0x7b19ef in QDR::initialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/jdanek/repos/skupper-router/tests/c_unittests/helpers.hpp:206
    #65 0x8974bc in operator() /home/jdanek/repos/skupper-router/tests/c_unittests/test_http_listener_connector.cpp:67
    #66 0x7f2ba3ee31e2 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xe31e2) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)
    #67 0x7f2ba3cac896 in start_thread (/lib64/libc.so.6+0x8e896) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #68 0x7f2ba3d336fb in clone3 (/lib64/libc.so.6+0x1156fb) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)

0x6040000aba78 is located 40 bytes inside of 44-byte region [0x6040000aba50,0x6040000aba7c)
freed by thread T1 here:
    #0 0x7f2ba54d7fb8 in __interceptor_free.part.0 (/lib64/libasan.so.8+0xd7fb8) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f2ba4c77b26 in _PyMem_RawFree (/lib64/libpython3.12d.so.1.0+0x277b26) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #2 0x7f2ba4c792a2 in PyObject_Free (/lib64/libpython3.12d.so.1.0+0x2792a2) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #3 0x7f2ba4cd7dcf in unicode_dealloc (/lib64/libpython3.12d.so.1.0+0x2d7dcf) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #4 0x7f2ba4c7778a in _Py_Dealloc (/lib64/libpython3.12d.so.1.0+0x27778a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #5 0x7f2ba4c50b77 in Py_DECREF.lto_priv.27 (/lib64/libpython3.12d.so.1.0+0x250b77) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #6 0x7f2ba4c50bcc in Py_XDECREF.lto_priv.25 (/lib64/libpython3.12d.so.1.0+0x250bcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #7 0x7f2ba4c533c8 in free_keys_object (/lib64/libpython3.12d.so.1.0+0x2533c8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #8 0x7f2ba4c524ca in dictkeys_decref (/lib64/libpython3.12d.so.1.0+0x2524ca) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #9 0x7f2ba4c575ba in PyDict_Clear (/lib64/libpython3.12d.so.1.0+0x2575ba) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #10 0x7f2ba4cc4c93 in clear_interned_dict (/lib64/libpython3.12d.so.1.0+0x2c4c93) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #11 0x7f2ba4d0136c in _PyUnicode_ClearInterned (/lib64/libpython3.12d.so.1.0+0x30136c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #12 0x7f2ba4e0da25 in finalize_interp_types (/lib64/libpython3.12d.so.1.0+0x40da25) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #13 0x7f2ba4e0db0c in finalize_interp_clear (/lib64/libpython3.12d.so.1.0+0x40db0c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #14 0x7f2ba4e0dc99 in Py_FinalizeEx (/lib64/libpython3.12d.so.1.0+0x40dc99) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #15 0x7f2ba4e0dd07 in Py_Finalize (/lib64/libpython3.12d.so.1.0+0x40dd07) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #16 0x5ccdb0 in qd_python_finalize /home/jdanek/repos/skupper-router/src/python_embedded.c:71
    #17 0x57ac8a in qd_dispatch_free /home/jdanek/repos/skupper-router/src/dispatch.c:394
    #18 0x879f06 in qd_dispatch_free /home/jdanek/repos/skupper-router/src/dispatch.c:374
    #19 0x879f06 in QDR::deinitialize(bool) const [clone .constprop.0] /home/jdanek/repos/skupper-router/tests/c_unittests/helpers.hpp:264
    #20 0x8a95be in operator() /home/jdanek/repos/skupper-router/tests/c_unittests/test_connection_manager_static.cpp:131
    #21 0x7f2ba3ee31e2 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xe31e2) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)

previously allocated by thread T1 here:
    #0 0x7f2ba54d92ef in malloc (/lib64/libasan.so.8+0xd92ef) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f2ba4c77a7d in _PyMem_RawMalloc (/lib64/libpython3.12d.so.1.0+0x277a7d) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #2 0x7f2ba4c791ab in PyObject_Malloc (/lib64/libpython3.12d.so.1.0+0x2791ab) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #3 0x7f2ba4cd6550 in PyUnicode_New (/lib64/libpython3.12d.so.1.0+0x2d6550) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #4 0x7f2ba4ce4c72 in unicode_decode_utf8 (/lib64/libpython3.12d.so.1.0+0x2e4c72) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #5 0x7f2ba4ce523f in PyUnicode_DecodeUTF8Stateful (/lib64/libpython3.12d.so.1.0+0x2e523f) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #6 0x7f2ba4cd8da1 in PyUnicode_FromString (/lib64/libpython3.12d.so.1.0+0x2d8da1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #7 0x7f2ba4d01125 in PyUnicode_InternFromString (/lib64/libpython3.12d.so.1.0+0x301125) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #8 0x7f2ba0a44b03 in CreateSwappedType (/usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312d-x86_64-linux-gnu.so+0xcb03) (BuildId: cdb5a61ddd026358017a72a3d3405dd431e31b72)
    #9 0x7f2ba0a4552c in PyCSimpleType_new (/usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312d-x86_64-linux-gnu.so+0xd52c) (BuildId: cdb5a61ddd026358017a72a3d3405dd431e31b72)
    #10 0x7f2ba4c958c3 in type_call (/lib64/libpython3.12d.so.1.0+0x2958c3) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #11 0x7f2ba4bf8792 in _PyObject_MakeTpCall (/lib64/libpython3.12d.so.1.0+0x1f8792) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #12 0x7f2ba4bf825c in _PyObject_FastCallDictTstate (/lib64/libpython3.12d.so.1.0+0x1f825c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #13 0x7f2ba4bf8389 in PyObject_VectorcallDict (/lib64/libpython3.12d.so.1.0+0x1f8389) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #14 0x7f2ba4d5afd1 in builtin___build_class__ (/lib64/libpython3.12d.so.1.0+0x35afd1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #15 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #16 0x7f2ba4bf7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #17 0x7f2ba4bf8b98 in PyObject_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8b98) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #18 0x7f2ba4d7b662 in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x37b662) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #19 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #20 0x7f2ba4d86a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #21 0x7f2ba4d63933 in PyEval_EvalCode (/lib64/libpython3.12d.so.1.0+0x363933) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #22 0x7f2ba4d5c678 in builtin_exec_impl (/lib64/libpython3.12d.so.1.0+0x35c678) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #23 0x7f2ba4d59a3a in builtin_exec (/lib64/libpython3.12d.so.1.0+0x359a3a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #24 0x7f2ba4c6df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #25 0x7f2ba4bf8909 in _PyVectorcall_Call (/lib64/libpython3.12d.so.1.0+0x1f8909) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #26 0x7f2ba4bf8cf1 in _PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8cf1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #27 0x7f2ba4bf8dcc in PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8dcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #28 0x7f2ba4d814dc in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x3814dc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #29 0x7f2ba4d61f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)

Thread T4 created by T0 here:
    #0 0x7f2ba5448956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f2ba3ee32b8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xe32b8) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)
    #2 0x7ca08e in DOCTEST_ANON_FUNC_29 /home/jdanek/repos/skupper-router/tests/c_unittests/test_http_listener_connector.cpp:65
    #3 0x78a350 in doctest::Context::run() /home/jdanek/repos/skupper-router/tests/c_unittests/doctest.h:7007
    #4 0x456aad in main /home/jdanek/repos/skupper-router/tests/c_unittests/c_unittests_main.cpp:76
    #5 0x7f2ba3c46149 in __libc_start_call_main (/lib64/libc.so.6+0x28149) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #6 0x7f2ba3c4620a in __libc_start_main_impl (/lib64/libc.so.6+0x2820a) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #7 0x45c874 in _start (/home/jdanek/repos/skupper-router/cmake-build-relwithdebinfo-asan/tests/c_unittests/c_unittests+0x45c874) (BuildId: 7638f1da81a091da1a78ee3ba7675e8ad027ba1d)

Thread T1 created by T0 here:
    #0 0x7f2ba5448956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f2ba3ee32b8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xe32b8) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)
    #2 0x7c9dde in DOCTEST_ANON_FUNC_26 /home/jdanek/repos/skupper-router/tests/c_unittests/test_connection_manager_static.cpp:71
    #3 0x78a350 in doctest::Context::run() /home/jdanek/repos/skupper-router/tests/c_unittests/doctest.h:7007
    #4 0x456aad in main /home/jdanek/repos/skupper-router/tests/c_unittests/c_unittests_main.cpp:76
    #5 0x7f2ba3c46149 in __libc_start_call_main (/lib64/libc.so.6+0x28149) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #6 0x7f2ba3c4620a in __libc_start_main_impl (/lib64/libc.so.6+0x2820a) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #7 0x45c874 in _start (/home/jdanek/repos/skupper-router/cmake-build-relwithdebinfo-asan/tests/c_unittests/c_unittests+0x45c874) (BuildId: 7638f1da81a091da1a78ee3ba7675e8ad027ba1d)

SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.8+0x6eb31) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c) in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x6040000ab780: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x6040000ab800: fa fa 00 00 00 00 00 00 fa fa fd fd fd fd fd fd
  0x6040000ab880: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x6040000ab900: fa fa 00 00 00 00 00 07 fa fa 00 00 00 00 00 00
  0x6040000ab980: fa fa 00 00 00 00 00 06 fa fa fd fd fd fd fd fd
=>0x6040000aba00: fa fa 00 00 00 00 00 fa fa fa fd fd fd fd fd[fd]
  0x6040000aba80: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
  0x6040000abb00: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x6040000abb80: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x6040000abc00: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x6040000abc80: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1716614==ABORTING

Process finished with exit code 1

and with pymalloc I got a different trace

=================================================================
==1717283==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f483bee2310 in thread T4
    #0 0x7f48420d7fb8 in __interceptor_free.part.0 (/lib64/libasan.so.8+0xd7fb8) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f4841877b26 in _PyMem_RawFree (/lib64/libpython3.12d.so.1.0+0x277b26) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #2 0x7f4841878e51 in PyMem_RawFree (/lib64/libpython3.12d.so.1.0+0x278e51) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #3 0x7f484187b787 in _PyObject_Free (/lib64/libpython3.12d.so.1.0+0x27b787) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #4 0x7f48418792a2 in PyObject_Free (/lib64/libpython3.12d.so.1.0+0x2792a2) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #5 0x7f4841855e50 in dictresize (/lib64/libpython3.12d.so.1.0+0x255e50) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #6 0x7f4841854949 in insertion_resize (/lib64/libpython3.12d.so.1.0+0x254949) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #7 0x7f4841854ce6 in insertdict (/lib64/libpython3.12d.so.1.0+0x254ce6) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #8 0x7f48418569c4 in _PyDict_SetItem_Take2 (/lib64/libpython3.12d.so.1.0+0x2569c4) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #9 0x7f4841856aa8 in PyDict_SetItem (/lib64/libpython3.12d.so.1.0+0x256aa8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #10 0x7f48418a4bf3 in add_subclass (/lib64/libpython3.12d.so.1.0+0x2a4bf3) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #11 0x7f48418a41a4 in type_ready_add_subclasses (/lib64/libpython3.12d.so.1.0+0x2a41a4) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #12 0x7f48418a45f4 in type_ready (/lib64/libpython3.12d.so.1.0+0x2a45f4) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #13 0x7f48418a47ad in PyType_Ready (/lib64/libpython3.12d.so.1.0+0x2a47ad) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #14 0x7f484189a983 in type_new_impl (/lib64/libpython3.12d.so.1.0+0x29a983) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #15 0x7f484189af7b in type_new (/lib64/libpython3.12d.so.1.0+0x29af7b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #16 0x7f483cbe0f52 in PyCFuncPtrType_new (/usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312d-x86_64-linux-gnu.so+0xdf52) (BuildId: cdb5a61ddd026358017a72a3d3405dd431e31b72)
    #17 0x7f48418958c3 in type_call (/lib64/libpython3.12d.so.1.0+0x2958c3) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #18 0x7f48417f8792 in _PyObject_MakeTpCall (/lib64/libpython3.12d.so.1.0+0x1f8792) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #19 0x7f48417f825c in _PyObject_FastCallDictTstate (/lib64/libpython3.12d.so.1.0+0x1f825c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #20 0x7f48417f8389 in PyObject_VectorcallDict (/lib64/libpython3.12d.so.1.0+0x1f8389) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #21 0x7f484195afd1 in builtin___build_class__ (/lib64/libpython3.12d.so.1.0+0x35afd1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #22 0x7f484197e62b in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x37e62b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #23 0x7f4841961f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #24 0x7f4841986a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #25 0x7f4841963933 in PyEval_EvalCode (/lib64/libpython3.12d.so.1.0+0x363933) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #26 0x7f484195c678 in builtin_exec_impl (/lib64/libpython3.12d.so.1.0+0x35c678) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #27 0x7f4841959a3a in builtin_exec (/lib64/libpython3.12d.so.1.0+0x359a3a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #28 0x7f484186df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #29 0x7f48417f8909 in _PyVectorcall_Call (/lib64/libpython3.12d.so.1.0+0x1f8909) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #30 0x7f48417f8cf1 in _PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8cf1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #31 0x7f48417f8dcc in PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8dcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #32 0x7f48419814dc in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x3814dc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #33 0x7f4841961f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #34 0x7f4841986a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #35 0x7f48417f8fdd in _PyFunction_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8fdd) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #36 0x7f48417f7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #37 0x7f48417fa545 in object_vacall (/lib64/libpython3.12d.so.1.0+0x1fa545) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #38 0x7f48417fa84a in PyObject_CallMethodObjArgs (/lib64/libpython3.12d.so.1.0+0x1fa84a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #39 0x7f48419dba6b in import_find_and_load (/lib64/libpython3.12d.so.1.0+0x3dba6b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #40 0x7f48419dbdb8 in PyImport_ImportModuleLevelObject (/lib64/libpython3.12d.so.1.0+0x3dbdb8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #41 0x7f4841988907 in import_name (/lib64/libpython3.12d.so.1.0+0x388907) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #42 0x7f4841976e96 in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x376e96) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #43 0x7f4841961f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #44 0x7f4841986a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #45 0x7f4841963933 in PyEval_EvalCode (/lib64/libpython3.12d.so.1.0+0x363933) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #46 0x7f484195c678 in builtin_exec_impl (/lib64/libpython3.12d.so.1.0+0x35c678) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #47 0x7f4841959a3a in builtin_exec (/lib64/libpython3.12d.so.1.0+0x359a3a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #48 0x7f484186df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #49 0x7f48417f8909 in _PyVectorcall_Call (/lib64/libpython3.12d.so.1.0+0x1f8909) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #50 0x7f48417f8cf1 in _PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8cf1) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #51 0x7f48417f8dcc in PyObject_Call (/lib64/libpython3.12d.so.1.0+0x1f8dcc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #52 0x7f48419814dc in _PyEval_EvalFrameDefault (/lib64/libpython3.12d.so.1.0+0x3814dc) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #53 0x7f4841961f81 in _PyEval_EvalFrame.lto_priv.1 (/lib64/libpython3.12d.so.1.0+0x361f81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #54 0x7f4841986a6c in _PyEval_Vector (/lib64/libpython3.12d.so.1.0+0x386a6c) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #55 0x7f48417f8fdd in _PyFunction_Vectorcall (/lib64/libpython3.12d.so.1.0+0x1f8fdd) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #56 0x7f48417f7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #57 0x7f48417fa545 in object_vacall (/lib64/libpython3.12d.so.1.0+0x1fa545) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #58 0x7f48417fa84a in PyObject_CallMethodObjArgs (/lib64/libpython3.12d.so.1.0+0x1fa84a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #59 0x7f48419dba6b in import_find_and_load (/lib64/libpython3.12d.so.1.0+0x3dba6b) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #60 0x7f48419dbdb8 in PyImport_ImportModuleLevelObject (/lib64/libpython3.12d.so.1.0+0x3dbdb8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #61 0x7f484195b1ee in builtin___import___impl (/lib64/libpython3.12d.so.1.0+0x35b1ee) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #62 0x7f48419591d8 in builtin___import__ (/lib64/libpython3.12d.so.1.0+0x3591d8) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #63 0x7f484186df9a in cfunction_vectorcall_FASTCALL_KEYWORDS (/lib64/libpython3.12d.so.1.0+0x26df9a) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #64 0x7f48417f7c81 in _PyObject_VectorcallTstate.lto_priv.3 (/lib64/libpython3.12d.so.1.0+0x1f7c81) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #65 0x7f48417f9595 in _PyObject_CallFunctionVa (/lib64/libpython3.12d.so.1.0+0x1f9595) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #66 0x7f48417f96e5 in PyObject_CallFunction (/lib64/libpython3.12d.so.1.0+0x1f96e5) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #67 0x7f48419dc3f5 in PyImport_Import (/lib64/libpython3.12d.so.1.0+0x3dc3f5) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #68 0x7f48419dae85 in PyImport_ImportModule (/lib64/libpython3.12d.so.1.0+0x3dae85) (BuildId: 3c61128b1fd63fa1904863ae8c5c9718b1cd55c4)
    #69 0x5e5700 in qd_python_setup /home/jdanek/repos/skupper-router/src/python_embedded.c:841
    #70 0x57b5a0 in qd_python_initialize /home/jdanek/repos/skupper-router/src/python_embedded.c:58
    #71 0x57b5a0 in qd_dispatch /home/jdanek/repos/skupper-router/src/dispatch.c:137
    #72 0x7b19ef in QDR::initialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/jdanek/repos/skupper-router/tests/c_unittests/helpers.hpp:206
    #73 0x8974bc in operator() /home/jdanek/repos/skupper-router/tests/c_unittests/test_http_listener_connector.cpp:67
    #74 0x7f4840ae31e2 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xe31e2) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)
    #75 0x7f48408ac896 in start_thread (/lib64/libc.so.6+0x8e896) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #76 0x7f48409336fb in clone3 (/lib64/libc.so.6+0x1156fb) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)

Address 0x7f483bee2310 is a wild pointer inside of access range of size 0x000000000001.
SUMMARY: AddressSanitizer: bad-free (/lib64/libasan.so.8+0xd7fb8) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c) in __interceptor_free.part.0
Thread T4 created by T0 here:
    #0 0x7f4842048956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: 7fcb7759bc17ef47f9682414b6d99732d6a6ab0c)
    #1 0x7f4840ae32b8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xe32b8) (BuildId: 43b5fe9c4fe0185772be4b57539be137c14a5c05)
    #2 0x7ca08e in DOCTEST_ANON_FUNC_29 /home/jdanek/repos/skupper-router/tests/c_unittests/test_http_listener_connector.cpp:65
    #3 0x78a350 in doctest::Context::run() /home/jdanek/repos/skupper-router/tests/c_unittests/doctest.h:7007
    #4 0x456aad in main /home/jdanek/repos/skupper-router/tests/c_unittests/c_unittests_main.cpp:76
    #5 0x7f4840846149 in __libc_start_call_main (/lib64/libc.so.6+0x28149) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #6 0x7f484084620a in __libc_start_main_impl (/lib64/libc.so.6+0x2820a) (BuildId: 788cdd41a15985bf8e0a48d213a46e07d58822df)
    #7 0x45c874 in _start (/home/jdanek/repos/skupper-router/cmake-build-relwithdebinfo-asan/tests/c_unittests/c_unittests+0x45c874) (BuildId: 7638f1da81a091da1a78ee3ba7675e8ad027ba1d)

==1717283==ABORTING

Process finished with exit code 1

Does this look like a Python bug? I'd be happy to attempt a reproducer if the backtraces are deemed sufficiently interesting and worth investigating.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

$ python3-debug -VV
Python 3.12.0 (main, Oct  2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)]
### Tasks
jiridanek commented 10 months ago

The reproducer is actually quite easy. It is enough to initialize and deinitialize Python interpreter twice in the same process.

#include "./qdr_doctest.hpp"
#include "./helpers.hpp"  // must come after ./qdr_doctest.hpp

#include <Python.h>

#include <thread>

TEST_CASE("Initialize and deinitialize Python twice")
{
    std::thread([]() {
        Py_Initialize();
        PyObject *m = PyImport_ImportModule("ctypes");
        Py_DECREF(m);
        PyGC_Collect();
        Py_Finalize();
    }).join();

    std::thread([]() {
        Py_Initialize();
        PyObject *m = PyImport_ImportModule("ctypes");
        Py_DECREF(m);
        PyGC_Collect();
        Py_Finalize();
    }).join();
}

I arrived at this by simplifying my original test at https://github.com/skupperproject/skupper-router/blob/cfc8b4c6892dafc5dd0c86682b4db87a488f2451/tests/c_unittests/test_router_startup.cpp#L25

sobolevn commented 10 months ago

cc @pablogsal

pablogsal commented 10 months ago

This looks like an issue in ctypes so we need to look at what changed from 3.11 to 3.12. @jiridanek can you try to bisect using your reproducer?

jiridanek commented 10 months ago

@sobolevn @pablogsal here's a suspected culprit from git-bisect

67807cfc87135fdce4992d38d2ffe3e44747e73b is the first bad commit
commit 67807cfc87135fdce4992d38d2ffe3e44747e73b
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Fri Nov 11 16:30:46 2022 -0700

    gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)

    The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject.  State for the "small block" allocator was stored in another 13.  That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057.  (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)

    https://github.com/python/cpython/issues/81057

 Include/internal/pycore_obmalloc.h                 |  689 +++++++++++++
 Include/internal/pycore_obmalloc_init.h            |   68 ++
 Include/internal/pycore_pymem.h                    |   35 +-
 Include/internal/pycore_pymem_init.h               |   85 ++
 Include/internal/pycore_runtime.h                  |    5 +
 Include/internal/pycore_runtime_init.h             |   10 +-
 Makefile.pre.in                                    |    3 +
 .../2022-11-07-14-16-59.gh-issue-81057.3uKlLQ.rst  |    3 +
 Objects/obmalloc.c                                 | 1070 ++++----------------
 PCbuild/pythoncore.vcxproj                         |    3 +
 PCbuild/pythoncore.vcxproj.filters                 |    9 +
 Python/pylifecycle.c                               |    2 +-
 Python/pystate.c                                   |    7 +-
 Tools/c-analyzer/cpython/globals-to-fix.tsv        |   15 +-
 14 files changed, 1104 insertions(+), 900 deletions(-)
 create mode 100644 Include/internal/pycore_obmalloc.h
 create mode 100644 Include/internal/pycore_obmalloc_init.h
 create mode 100644 Include/internal/pycore_pymem_init.h
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2022-11-07-14-16-59.gh-issue-81057.3uKlLQ.rst

So the culprit seems to be from this PR

from two years ago, by @ericsnowcurrently

Full git-bisect log:

``` jdanek@fedora:~/repos/cpython$ git bisect start good v3.11.0 bad main status: waiting for both good and bad commits jdanek@fedora:~/repos/cpython$ git bisect good v3.11.0 status: waiting for bad commit, 1 good commit known jdanek@fedora:~/repos/cpython$ git bisect bad main Bisecting: a merge base must be tested [8d32a5c8c4e9c90b0a21e8b2fa0cd567465ced2b] Update the SOURCE_URI in pyspecific.py to point to the new branch jdanek@fedora:~/repos/cpython$ git bisect good No testable commit found. Maybe you started with bad path arguments? jdanek@fedora:~/repos/cpython$ git bisect start Previous HEAD position was 8d32a5c8c4 Update the SOURCE_URI in pyspecific.py to point to the new branch HEAD is now at deaf509e8f Python 3.11.0 status: waiting for both good and bad commits jdanek@fedora:~/repos/cpython$ git bisect good 8d32a5c8c4e9c90b0a21e8b2fa0cd567465ced2b status: waiting for bad commit, 1 good commit known jdanek@fedora:~/repos/cpython$ git bisect bad main Bisecting: 3404 revisions left to test after this (roughly 12 steps) [accb417c338630ac6e836a5c811a89d54a3cd1d3] Replace Netlify with Read the Docs build previews (#103843) jdanek@fedora:~/repos/cpython$ git bisect skip Bisecting: 3402 revisions left to test after this (roughly 12 steps) [f186557dc3e77495ba65b9b7e492ab6ddb7cecc3] gh-103977: compile re expressions in platform.py only if required (#103981) jdanek@fedora:~/repos/cpython$ git bisect skip Bisecting: 3402 revisions left to test after this (roughly 12 steps) [91441bf7cbaefbd328ee7efa59a06e661f61542e] Docs: fix formatting in 2023-06-09-12-59-18 NEWS item (#105607) jdanek@fedora:~/repos/cpython$ git bisect bad Bisecting: 2026 revisions left to test after this (roughly 11 steps) [8dbe08eb7c807f484fe9870f5b7f5ae2881fd966] gh-99240: Fix double-free bug in Argument Clinic str_converter generated code (GH-99241) jdanek@fedora:~/repos/cpython$ git bisect bad Bisecting: 1012 revisions left to test after this (roughly 10 steps) [2fa03b1b0708d5d74630c351ec9abd2aac7550da] gh-95231: Disable md5 & crypt modules if FIPS is enabled (GH-94742) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 505 revisions left to test after this (roughly 9 steps) [187e853690908ca2af19a0701ca7529b43d05df9] gh-83940: os docs: Improve wording for getenv/getenvb (#98113) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 252 revisions left to test after this (roughly 8 steps) [cff1c20667f668f20be9377ba2bd8d1dc9d56bec] gh-94199: Remove ssl.wrap_socket() documentation (#99023) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 125 revisions left to test after this (roughly 7 steps) [504e12272b19395a3c59631a20a94e3323af78e5] GH-81057: Remove static state from arraymodule (#99409) jdanek@fedora:~/repos/cpython$ git bisect bad Bisecting: 62 revisions left to test after this (roughly 6 steps) [c32bc1bffd9d63ede0d0505abab983247a3ad0c6] gh-99124: use concrete exception types in `test_builtin` (#99125) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 30 revisions left to test after this (roughly 5 steps) [1e197e63e21f77b102ff2601a549dda4b6439455] GH-96421: Insert shim frame on entry to interpreter (GH-96319) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 14 revisions left to test after this (roughly 4 steps) [67b4d2772c5124b908f8ed9b13166a79bbeb88d2] gh-98086: Now ``patch.dict`` can decorate async functions (#98095) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 6 revisions left to test after this (roughly 3 steps) [dd36b71fa6164ebba5d94bb4a24eac43b1c54906] gh-81057: Move the Extension Modules Cache to _PyRuntimeState (gh-99355) jdanek@fedora:~/repos/cpython$ git bisect good Bisecting: 3 revisions left to test after this (roughly 2 steps) [67807cfc87135fdce4992d38d2ffe3e44747e73b] gh-81057: Move the Allocators to _PyRuntimeState (gh-99217) jdanek@fedora:~/repos/cpython$ git bisect bad Bisecting: 0 revisions left to test after this (roughly 1 step) [55c96e8053689c29ae28a9d2117ae37934eace68] gh-99305: Speed up secrets.token_hex() ~2x (#99306) jdanek@fedora:~/repos/cpython$ git bisect good 67807cfc87135fdce4992d38d2ffe3e44747e73b is the first bad commit ```
ericsnowcurrently commented 4 months ago

This should be resolved in main and in the 3.13 branch, since we did the work to isolate _ctypes to each interpreter. Fixing this in 3.12 will be a different matter.