[Windows] Embedded Python Memory Leaks

[AlexSoft73]

Hi dear Python team,

I am a building a c/c++ application for general purpose and I would like to have in the application the possibility of running “Python” scripting code. Python has proved to be a good programming language to deal with heavy Math as in AI.

Now when I embed Python in my application things go perfect, except for memory leaks! As a sample by just issuing the following code leaves memory leaks of around (2Mb):

#define PY_SSIZE_T_CLEAN
#define Py_LIMITED_API 0x03000000
#include <Python.h>
#pragma comment (lib,"python3.lib")

int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow)
{
  Py_InitializeEx(0); //For Embeded Python
  Py_FinalizeEx();
  ..More code..
  return 0;
}

Is there something I must do to cause Python to release all resources allocated

[vstinner]

The refcount tracking and allocated block tracking don't pick those up since they only cover pymalloc and actual Python objects

I tried once to account PyMem_RawMalloc() in "allocated blocks": https://bugs.python.org/issue26850

But with this change, Python said that it leaked memory. I gave up in 2016. In the meanwhile, many "leaks" were fixed: Python now releases most memory at exit.

[AlexSoft73]

Thanks ncoghlan for your replay and interest in getting Python improved,

I believe it all lies in the Python startup global configuration data not being released in Py_Finalize(); See if this rings you a bell, I've been tracing the allocations.

The following stack frame is one where I see (262168 ==> 256Kb) allocated but never released, happening in file: obmalloc.c function: static Py_ALWAYS_INLINE arena_map_bot_t arena_map_get(block p, int create) Line: arena_map_mid_t *n = PyMem_RawCalloc(1, sizeof(arena_map_mid_t));

Stack frame:

python312_d.dll!_PyMem_RawCalloc(void _unused_ctx, unsigned int64 nelem, unsigned int64 elsize) Line 148 C python312_d.dll!_PyMem_DebugRawAlloc(int use_calloc, void ctx, unsigned int64 nbytes) Line 2573 C python312_d.dll!_PyMem_DebugRawCalloc(void ctx, unsigned int64 nelem, unsigned int64 elsize) Line 2619 C python312_d.dll!PyMem_RawCalloc(unsigned int64 nelem, unsigned int64 elsize) Line 664 C python312_d.dll!arena_map_get(unsigned char p, int create) Line 1527 C python312_d.dll!arena_map_mark_used(unsigned int64 arena_base, int is_used) Line 1582 C python312_d.dll!new_arena() Line 1712 C python312_d.dll!allocate_from_new_pool(unsigned int size) Line 1892 C python312_d.dll!pymalloc_alloc(void _unused_ctx, unsigned __int64 nbytes) Line 2056 C python312_d.dll!_PyObject_Malloc(void ctx, unsigned int64 nbytes) Line 2066 C python312_d.dll!_PyMem_DebugRawAlloc(int use_calloc, void * ctx, unsigned int64 nbytes) Line 2576 C python312_d.dll!_PyMem_DebugRawMalloc(void ctx, unsigned __int64 nbytes) Line 2610 C python312_d.dll!_PyMem_DebugMalloc(void ctx, unsigned int64 nbytes) Line 2762 C python312_d.dll!PyObject_Malloc(unsigned int64 size) Line 781 C python312_d.dll!gc_alloc(unsigned int64 basicsize, unsigned int64 presize) Line 2286 C python312_d.dll!_PyObject_GC_New(_typeobject tp) Line 2301 C python312_d.dll!new_dict(_dictkeysobject keys, _dictvalues values, __int64 used, int free_values_on_failure) Line 737 C python312_d.dll!PyDict_New() Line 842 C python312_d.dll!PyUnicode_InternInPlace(_object p) Line 14669 C python312_d.dll!_PyUnicode_InitStaticStrings() Line 1433 C python312_d.dll!_PyUnicode_InitGlobalObjects(_is interp) Line 14610 C python312_d.dll!pycore_init_global_objects(_is interp) Line 682 C python312_d.dll!pycore_interp_init(_ts tstate) Line 826 C python312_d.dll!pyinit_config(pyruntimestate runtime, _ts tstate_p, const PyConfig config) Line 898 C python312_d.dll!pyinit_core(pyruntimestate runtime, const PyConfig src_config, _ts tstate_p) Line 1061 C python312_d.dll!Py_InitializeFromConfig(const PyConfig * config) Line 1256 C python312_d.dll!Py_InitializeEx(int install_sigs) Line 1294 C

[AlexSoft73]

Hi @ncoghlan and thanks again

About this comment,

There are a handful of unmatched PyMem_RawMalloc calls (related to _Py_path_config as @vstinner noted above), but those don't get repeated (later interpreter instances reuse the previous allocations) so the total memory use doesn't grow over time.

I consider the above partially true, partially false, the thing lies here (This is a subject I wanted to talk about after we got resolved the memory issues to go one step at e time).

There is a difference when we use python3.dll or python3XX.dll, The one most important to me is:

With python3.dll is posible late binding, hence I embed python functionality use it and when done with python release the pythin3.dll and I am finish with python till I need it again. It is here where I'd like to see python releasing all the resources it took from the OS. Why? well next time the same application (still running) needs to use Python, then load Python3.dll again, all those global variables left by previous load become memory leaks.

Please, if you don't understand something of the above, let me know, it is a bit tricky ti explain to me. Thanks in advance,