About todos: GC and pymalloc

[JunyiXie]

Half a year ago, I implemented a similar change to parallelize the sub-interpreters in the company's internal project. I noticed that there are related to-dos in the Dev documentation. My handling of these issues is:

• Each sub-interpreter holds a garbage collection.
• Each sub-interpreter holds a memory pool. I look forward to your changes being merged into 3.10, so that I don’t have to maintain my own custom changes based on version 3.8.

https://pythondev.readthedocs.io/subinterpreters.html

TODO list for per-interpreter GIL

Search for Subinterpreters issues at bugs.python.org.

Meta issue: per-interpreter GIL.

Effects of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro:

Good things! Per-interpreter GIL!!! Use a TSS to get the current Python thread state (‘tstate’) _xxsubinterpreters.run_string() releases the GIL to run a subinterprer Bad things :-( (mostly workarounds waiting for a real fix) Disable pymalloc in preconfig.c: force malloc (or malloc_debug) allocator. Don’t run GC collections in subinterpreters (see gc_collect_main).

[jakirkham]

I think the GC already got moved into subinterpreters ( https://github.com/ericsnowcurrently/multi-core-python/issues/29 ). FWIU fixing pymalloc is probably more complicated due to the GIL, which is tracked here ( https://github.com/ericsnowcurrently/multi-core-python/issues/30 ). There is a workaround to use malloc as suggested, which was included in PR ( https://github.com/python/cpython/pull/19926 ).

[JunyiXie]

At first, I simply closed pymalloc, but I found that in many scenarios, this will seriously affect the performance of python execution. If the implementation of a multi-sub interpreter has too much impact on the performance of a single interpreter, I think the multi-sub interpreter does not have much practical value.

So I started to adapt the pymalloc multi-sub interpreter My modification method is:

1. Use Malloc to allocate before creating the main interpreter.
2. The interpreter uses its own pymalloc memory pool after creation
3. Maintain a linked list of memory pools, memory pools can be reused

About 3. Maintain a linked list of memory pools, memory pools can be reused I found some memory leaks in the interpreter. If the memory pool is released directly when the interpreter is destroyed, there will be some problems next time the interpreter is created and run. So I bypassed this problem by reusing the memory pool.