Open jiridanek opened 10 months ago
The reproducer is actually quite easy. It is enough to initialize and deinitialize Python interpreter twice in the same process.
#include "./qdr_doctest.hpp"
#include "./helpers.hpp" // must come after ./qdr_doctest.hpp
#include <Python.h>
#include <thread>
TEST_CASE("Initialize and deinitialize Python twice")
{
std::thread([]() {
Py_Initialize();
PyObject *m = PyImport_ImportModule("ctypes");
Py_DECREF(m);
PyGC_Collect();
Py_Finalize();
}).join();
std::thread([]() {
Py_Initialize();
PyObject *m = PyImport_ImportModule("ctypes");
Py_DECREF(m);
PyGC_Collect();
Py_Finalize();
}).join();
}
I arrived at this by simplifying my original test at https://github.com/skupperproject/skupper-router/blob/cfc8b4c6892dafc5dd0c86682b4db87a488f2451/tests/c_unittests/test_router_startup.cpp#L25
cc @pablogsal
This looks like an issue in ctypes
so we need to look at what changed from 3.11 to 3.12. @jiridanek can you try to bisect using your reproducer?
@sobolevn @pablogsal here's a suspected culprit from git-bisect
67807cfc87135fdce4992d38d2ffe3e44747e73b is the first bad commit
commit 67807cfc87135fdce4992d38d2ffe3e44747e73b
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date: Fri Nov 11 16:30:46 2022 -0700
gh-81057: Move the Allocators to _PyRuntimeState (gh-99217)
The global allocators were stored in 3 static global variables: _PyMem_Raw, _PyMem, and _PyObject. State for the "small block" allocator was stored in another 13. That makes a total of 16 global variables. We are moving all 16 to the _PyRuntimeState struct as part of the work for gh-81057. (If PEP 684 is accepted then we will follow up by moving them all to PyInterpreterState.)
https://github.com/python/cpython/issues/81057
Include/internal/pycore_obmalloc.h | 689 +++++++++++++
Include/internal/pycore_obmalloc_init.h | 68 ++
Include/internal/pycore_pymem.h | 35 +-
Include/internal/pycore_pymem_init.h | 85 ++
Include/internal/pycore_runtime.h | 5 +
Include/internal/pycore_runtime_init.h | 10 +-
Makefile.pre.in | 3 +
.../2022-11-07-14-16-59.gh-issue-81057.3uKlLQ.rst | 3 +
Objects/obmalloc.c | 1070 ++++----------------
PCbuild/pythoncore.vcxproj | 3 +
PCbuild/pythoncore.vcxproj.filters | 9 +
Python/pylifecycle.c | 2 +-
Python/pystate.c | 7 +-
Tools/c-analyzer/cpython/globals-to-fix.tsv | 15 +-
14 files changed, 1104 insertions(+), 900 deletions(-)
create mode 100644 Include/internal/pycore_obmalloc.h
create mode 100644 Include/internal/pycore_obmalloc_init.h
create mode 100644 Include/internal/pycore_pymem_init.h
create mode 100644 Misc/NEWS.d/next/Core and Builtins/2022-11-07-14-16-59.gh-issue-81057.3uKlLQ.rst
So the culprit seems to be from this PR
from two years ago, by @ericsnowcurrently
Full git-bisect log:
This should be resolved in main and in the 3.13 branch, since we did the work to isolate _ctypes
to each interpreter. Fixing this in 3.12 will be a different matter.
Crash report
What happened?
I have a binary which is embedding python. I want to test this binary with address sanitizer, and leak sanitizer. Regular python3.12 build contained in Fedora 39 causes lsan to report leaks even with
PYTHONMALLOC=malloc_debug
, which with previous versions gave me clean run.Therefore, I tried
python3-debug
, which is a debug build of python packaged in Fedora. With that, I hitand I was playing around a bit. When I hit the mentioned issue, I was using PYTHONMALLOC="malloc_debug". I wanted to try other values for this variable. With
pymalloc_debug
, I got the same assertion.With
malloc
, I gotand with
pymalloc
I got a different traceDoes this look like a Python bug? I'd be happy to attempt a reproducer if the backtraces are deemed sufficiently interesting and worth investigating.
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line: