AlexSoft73
commented
2 years ago Thanks in advance
Closed AlexSoft73 closed 1 year ago
AlexSoft73
commented
2 years ago Thanks in advance
sweeneyde
commented
2 years ago I believe this is fixed in Python 3.11, and was tracked by https://github.com/python/cpython/issues/44470. I doubt there would be a backport, since by my understanding, much of the work to fix this was rather involved.
cc @vstinner
AlexSoft73
commented
2 years ago I am actually using Python 3.11 and it is happening. It will be of a tremendous help to see that fixed.
vstinner
commented
2 years ago ..More code..
Can you please provide a full reproducer? Like a whole C file which reproduces the issue? If possible, a regular main() function, not a "WINAPI wWinMain" function.
Can you reproduce the issue on other operating systems? How do you link your program?
AlexSoft73
commented
2 years ago Thanks a lot for your support.
I am assuming your environment of work is Unix/Linux, I cannot speak about memory leaks in Unix environment since I am not working performing my tests in this unix environment.
The code I sent is what it is, it is that simple:
//This is the main entry point for a C/C++ code copiled for Windows 64bit in Microsoft Visual Studio 2029 //int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow) int main(...) { Py_InitializeEx(0); Py_FinalizeEx(); Py_InitializeEx(0); Py_FinalizeEx(); return 0; }
Put a comment in that line and add the main prototype your compliler/linker likes. Trace that code and you will see how memory leaks.
I repeat, this memory leaks might not happen in Unix/Linux environment, I suggest testing this code in windows enviroment and Python 3.11
Thanks a million, Alexei
PD: If you have and have permission to share the original code for Py_InitializeEx and Py_FinalizeEx send the code to me and I will help you in troubleshooting this code for Windows
De: Victor Stinner @.> Enviado: lunes, 19 de septiembre de 2022 09:51 p. m. Para: python/cpython @.> CC: AlexSoft73 @.>; Author @.> Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
..More code..
Can you please provide a full reproducer? Like a whole C file which reproduces the issue? If possible, a regular main() function, not a "WINAPI wWinMain" function.
Can you reproduce the issue on other operating systems? How do you link your program?
— Reply to this email directly, view it on GitHubhttps://github.com/python/cpython/issues/96853#issuecomment-1251600183, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3C7TGERMLZNFLTC3MUHF2DV7DN6XANCNFSM6AAAAAAQNW7ZHA. You are receiving this because you authored the thread.Message ID: @.***>
AlexSoft73
commented
2 years ago Thanks for your reply,
Well, i am testing my code with Python 3.11 (64bits), the latest version available and it is leaking memory.
I read the whole #44470 and could not find even the reason for that. if you have the code and have permissio to share it, send me the source code for Py_InitializeEx and Py_FinalizeEx to see If I can help you in troubleshooting the memory leaks.
Thanks in advance, Alexei
De: Dennis Sweeney @.> Enviado: viernes, 16 de septiembre de 2022 09:03 p. m. Para: python/cpython @.> CC: AlexSoft73 @.>; Author @.> Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
I believe this is fixed in Python 3.11, and was tracked by #44470https://github.com/python/cpython/issues/44470. I doubt there would be a backport, since by my understanding, much of the work to fix this was rather involved.
cc @vstinnerhttps://github.com/vstinner
— Reply to this email directly, view it on GitHubhttps://github.com/python/cpython/issues/96853#issuecomment-1249815679, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3C7TGDYEICQQOT5OXTTTT3V6TODNANCNFSM6AAAAAAQNW7ZHA. You are receiving this because you authored the thread.Message ID: @.***>
da-woods
commented
2 years ago It's open source - you don't need permission to see the code for PyInitializeEx and Py_FinalizeEx. You can read it here https://github.com/python/cpython/blob/4e4bfffe2ded7339663945ad4c494db0e88ec96c/Python/pylifecycle.c#L1274
AlexSoft73
commented
2 years ago Thanks a lot for the info, I am working on it to see if I can help you guys.
AlexSoft73
commented
2 years ago I cannot seem to find all files references in "pylifecycle.c" Where are these files:
ericsnowcurrently
commented
2 years ago They are in Include/internal.
AlexSoft73
commented
2 years ago Hi Python team,
I'd like to compile the python source code in VS2019 to try and debug the initialization process, hence be able to signal what is not release when python is called to finalized. Is there an easy way of getting all files (.h,.c,.cpp.,...) required for this
da-woods
commented
2 years ago @AlexSoft73 there's some information about how to download and compile Python at the top of this page https://devguide.python.org/
AlexSoft73
commented
1 year ago Yes, I saw it, I am trying to do it to find the reason for those memory leaks and let you knoqw guys
thanks
De: da-woods @.> Enviado: sábado, 24 de septiembre de 2022 06:36 a. m. Para: python/cpython @.> CC: AlexSoft73 @.>; Mention @.> Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
@AlexSoft73https://github.com/AlexSoft73 there's some information about how to download and compile Python at the top of this page https://devguide.python.org/
— Reply to this email directly, view it on GitHubhttps://github.com/python/cpython/issues/96853#issuecomment-1256880528, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3C7TGAH5PVUCRE7MYIYQQDV72ONDANCNFSM6AAAAAAQNW7ZHA. You are receiving this because you were mentioned.Message ID: @.***>
AlexSoft73
commented
1 year ago For some reason I am having no luck in this:
Git Error:
ssh: connect to host github.com port 22: Connection timed out fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
I am behind a proxy
De: da-woods @.> Enviado: sábado, 24 de septiembre de 2022 06:36 a. m. Para: python/cpython @.> CC: AlexSoft73 @.>; Mention @.> Asunto: Re: [python/cpython] Embedded Python Memory Leaks (Issue #96853)
@AlexSoft73https://github.com/AlexSoft73 there's some information about how to download and compile Python at the top of this page https://devguide.python.org/
— Reply to this email directly, view it on GitHubhttps://github.com/python/cpython/issues/96853#issuecomment-1256880528, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A3C7TGAH5PVUCRE7MYIYQQDV72ONDANCNFSM6AAAAAAQNW7ZHA. You are receiving this because you were mentioned.Message ID: @.***>
AlexSoft73
commented
1 year ago Hi dear python team,
when compiling in VS2019 I get the following error:
[error] 2>D:\Python\Source\cpython-main\Modules_ctypes\callproc.c(82,10): fatal error C1083: Cannot open include file: 'ffi.h': No such file or directory [/error]
I searched for this file I I cannot locate it, where can I find it please
thanks in advance
AlexSoft73
commented
1 year ago Good day Python Team,
I cannot get the python312.dll built (this is my file of interest to troubleshoot the memory leaks)
Thanks in advance, Alexei
AlexSoft73
commented
1 year ago AlexSoft73
commented
1 year ago Hi Python team,
I manage to get the project compile in MSVS2019.
Thanks in advance, Alexei
AlexSoft73
commented
1 year ago Understanding python when embedded.
Please, answer the following question
A module is an isolated python execution unit? Meaning each module has its own set of variables?
thanks in advance Alexei
da-woods
commented
1 year ago A module is an isolated python execution unit? Meaning each module has its own set of variables?
Each module has it's own namespace. However variables might be shared between a module. If module B does from A import x then the variable x is shared between A and B.
AlexSoft73
commented
1 year ago Yes, I understand the import part, if you import means you can access what you imported, ie:
import moduleA as mA (suposing moduleA insde defines a global variable AA) you can access this way mA.AA
Am I right?
AlexSoft73
commented
1 year ago da-woods
commented
1 year ago I've used embedded Python before but it's been a while since I last set it up.
AlexSoft73
commented
1 year ago Hi Python Team
Thanks is advance to all those that have helped me with my bigining in this forum.
============= Phase -1 Call 24 to cmalloc (requesting 262168 bytes)=================
I know when using python from as an application these memory leaks might have no relevance since the OS will free memory for you upon application is finished. Now when embedding Python we have a different story, especially if we are using Python as child threads of the main application thread.
The reason I am asking is, I am no expert in the internal working of Python, I am just pursuing memory leaks and trying to understand why it is happening to try and solve the problem.
============================================== Offending function _PyObject_GC_New in module module "gcmodule.c"
[python] _PyObject_GC_New(PyTypeObject tp) { size_t presize = _PyType_PreHeaderSize(tp); PyObject op = gc_alloc(_PyObject_SIZE(tp), presize); <---- This object (op) is never released when Py_finalize is called. if (op == NULL) { return NULL; } _PyObject_Init(op, tp); return op; } [/python]
================= Stack trace ===================
python312_d.dll!_PyObject_GC_New(_typeobject tp) Line 2302 C python312_d.dll!new_dict(_dictkeysobject keys, _dictvalues values, __int64 used, int free_values_on_failure) Line 737 C python312_d.dll!PyDict_New() Line 842 C python312_d.dll!PyUnicode_InternInPlace(_object p) Line 14669 C python312_d.dll!_PyUnicode_InitStaticStrings() Line 1433 C python312_d.dll!_PyUnicode_InitGlobalObjects(_is interp) Line 14610 C python312_d.dll!pycore_init_global_objects(_is interp) Line 682 C python312_d.dll!pycore_interp_init(_ts tstate) Line 826 C python312_d.dll!pyinit_config(pyruntimestate runtime, _ts tstate_p, const PyConfig config) Line 898 C python312_d.dll!pyinit_core(pyruntimestate runtime, const PyConfig src_config, _ts tstate_p) Line 1061 C python312_d.dll!Py_InitializeFromConfig(const PyConfig * config) Line 1256 C python312_d.dll!Py_InitializeEx(int install_sigs) Line 1294 C
AlexSoft73
commented
1 year ago Hi Python team,
In
finalize_restore_builtins(tstate); --> Why restore if we are Finalizing Python
This won't destroy the builtins
Thanks in advance Alexei
AlexSoft73
commented
1 year ago The sample code is as simple as
#include <Python.h>
int main(...)
{
//Step-1
Py_InitializeEx(0);
Py_FinalizeEx();
Step-2
Py_InitializeEx(0);
Py_FinalizeEx();
Step-2
Py_InitializeEx(0);
Py_FinalizeEx();
return 0;
}You will see how between this steps memory used is increasing. Usually while using the Python interpreter user don't get to see this since when the Python interpreter application calls Py_FinalizeEx(); and then finish, hence the OS frees whatever memory is unreleased
Thanks, Alexei
ncoghlan
commented
1 year ago While the init/finalise cycles in https://github.com/python/cpython/blob/main/Programs/_testembed.c are similar to the given example, one difference is that the embedding tests migrated at some point to using Py_InitializeFromConfig directly rather than continuing to use the Py_InitializeEx compatibility API.
Comparing the two initialisation functions, one discrepancy I see is that _testembed_Py_Initialize (https://github.com/python/cpython/blob/b0f89cb4311b696f875e58f14258ce315be09bce/Programs/_testembed.c#L76) is calling PyConfig_ClearConfig while Py_InitializeEx (https://github.com/python/cpython/blob/4e4bfffe2ded7339663945ad4c494db0e88ec96c/Python/pylifecycle.c#L1274) never clears its local config struct.
That doesn't seem right given the PyConfig docs say "When done, the PyConfig_Clear() function must be used to release the configuration memory."
AlexSoft73
commented
1 year ago Thanks a lot ncoghlan for your reply,
What I can see is that using this initialization method might do what need. One Caveat is: this function is not part of the Python ABI Function set, meaning this, I might lose compatibility among Python versions.
See, I really have concerns with compatibility, in fact I don't do early binding with Python libraries, I use late binding for 2 main reasons: 1- The application might not use Python at all in some circumstances, hence why load python stuff. 2- Let the user update python (as long as ABI set remains promised) and still be able to use Python without recompile the host application.
Again, thanks
AlexSoft73
commented
1 year ago Hi ncoghlan
Please, can you show me where are the instruction to format what i write, let's say c/c++ code or python code...
thanks
ncoghlan
commented
1 year ago @AlexSoft73 As in you'd like to see if using Py_InitializeFromConfig instead of Py_InitializeEx eliminates the detected leak? That experiment is a matter of using the code from https://docs.python.org/3/c-api/init_config.html#example but dropping the parts that customise the config away from the defaults:
// Start Py_InitializeEx replacement...
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
// Do something more graceful than terminating the entire process (which is what `Py_InitializeEx` will do if this fails)
}
PyConfig_Clear(&config); // This is the step that appears to be missing from `Py_InitializeEx`
// End of Py_InitializeEx replacement...If that change doesn't eliminate the leak you're seeing, then that's useful information as well: it means this isn't the problem, and the issue lies elsewhere (perhaps somewhere Windows specific)
vstinner
commented
1 year ago Now when I embed Python in my application things go perfect, except for memory leaks! As a sample by just issuing the following code leaves memory leaks of around (2Mb): (...)
How do you measure the memory? I'm not used to Windows development. On Linux, I check for lines started with Vm in /proc/pid/status:
$ echo $$
21584
$ grep ^Vm /proc/21584/status
VmPeak: 224772 kB
VmSize: 224772 kB
(....)About your comment:
============= Phase -1 Call 24 to cmalloc (requesting 262168 bytes)=================
(...)
==============================================
Offending function _PyObject_GC_New in module module "gcmodule.c"
(...)Where does it come from? Is it the output of a tool?
Or did you write it yourself?
I am actually using Python 3.11 and it is happening. It will be of a tremendous help to see that fixed.
Python 3.11.0 final is not released yet. Which exact version did you test?
vstinner
commented
1 year ago I did a test on Windows. I applied this patch:
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index e5b138ce84..fa196a3a20 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -158,6 +158,8 @@ PyInit_embedded_ext(void)
// multiple times.
static int test_repeated_init_exec(void)
{
+extern Py_ssize_t _Py_GetAllocatedBlocks(void);
+
if (main_argc < 3) {
fprintf(stderr, "usage: %s test_repeated_init_exec CODE\n", PROGRAM);
exit(1);
@@ -174,6 +176,8 @@ static int test_repeated_init_exec(void)
if (err) {
return 1;
}
+ printf("loop #%i: after Py_Finalize(): #refs=%zd, #blocks=%zd\n",
+ i, _Py_GetRefTotal(), _Py_GetAllocatedBlocks());
}
return 0;
}
Rebuilt Python and ran:
vstinner@WIN C:\victor\python\3.11>PCbuild\amd64\_testembed_d.exe test_repeated_init_exec pass
--- Loop #1 ---
loop #1: after Py_Finalize(): #refs=0, #blocks=0
--- Loop #2 ---
loop #2: after Py_Finalize(): #refs=0, #blocks=0
--- Loop #3 ---
loop #3: after Py_Finalize(): #refs=0, #blocks=0
--- Loop #4 ---
loop #4: after Py_Finalize(): #refs=0, #blocks=0I tested 3.11 and main branches: I get the same output. There is no leak according to _Py_GetRefTotal() and _Py_GetAllocatedBlocks(): memory allocated with Python memory allocators.
These functions don't account memory allocated in the "raw" memory allocation domain, functions like PyMem_RawMalloc(). For example, the "raw" domain is used by Python startup.
But your stack trace contains _PyObject_GC_New() which uses the "object" memory allocation domain: the PyObject_Malloc() function.
vstinner
commented
1 year ago On Linux, it looks like Py_Initialize()+Py_Finalize() called in a loop leaks...
$ ./Programs/_testembed test_repeated_init_exec 'import os; os.system(f"grep ^VmRSS /proc/{os.getpid()}/status")'
--- Loop #1 ---
VmRSS: 10136 kB
--- Loop #2 ---
VmRSS: 10564 kB
--- Loop #3 ---
VmRSS: 10624 kB
--- Loop #4 ---
VmRSS: 10680 kB... but if I change _testembed.c to loop more, in fact, the memory usage is stable (the leak was a false alarm):
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d635c5a4ab..fd61a4f26a 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -31,7 +31,7 @@ char **main_argv;
/* Use path starting with "./" avoids a search along the PATH */
#define PROGRAM_NAME L"./_testembed"
-#define INIT_LOOPS 4
+#define INIT_LOOPS 40
// Ignore Py_DEPRECATED() compiler warnings: deprecated functions are
// tested on purpose here.Output:
--- Loop #1 ---
VmRSS: 9984 kB
--- Loop #2 ---
VmRSS: 10420 kB
--- Loop #3 ---
VmRSS: 10496 kB
--- Loop #4 ---
VmRSS: 10560 kB
--- Loop #5 ---
VmRSS: 10608 kB
(...)
--- Loop #20 ---
VmRSS: 10712 kB
(...)
--- Loop #37 ---
VmRSS: 10712 kB
--- Loop #38 ---
VmRSS: 10712 kB
--- Loop #39 ---
VmRSS: 10712 kB
--- Loop #40 ---
VmRSS: 10708 kBI also ran Valgrind on the main branch:
PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./Programs/_testembed test_repeated_init_exec 'pass'Valgrind finds a leak of 1 kB:
==28543== HEAP SUMMARY:
==28543== in use at exit: 1,056 bytes in 6 blocks
==28543== total heap usage: 113,234 allocs, 113,228 frees, 15,378,972 bytes allocated
==28543==
==28543== 68 bytes in 1 blocks are possibly lost in loss record 1 of 6
==28543== at 0x484586F: malloc (vg_replace_malloc.c:381)
==28543== by 0x55193B: _PyMem_RawMalloc (obmalloc.c:101)
==28543== by 0x55449D: _PyMem_DebugRawAlloc (obmalloc.c:2509)
==28543== by 0x55456E: _PyMem_DebugRawMalloc (obmalloc.c:2542)
==28543== by 0x552598: PyMem_RawMalloc (obmalloc.c:587)
==28543== by 0x5527C5: _PyMem_RawWcsdup (obmalloc.c:668)
==28543== by 0x67D917: _PyPathConfig_UpdateGlobal (pathconfig.c:159)
==28543== by 0x69035E: interpreter_update_config (pylifecycle.c:476)
==28543== by 0x6921A6: init_interp_main (pylifecycle.c:1117)
==28543== by 0x6927C5: pyinit_main (pylifecycle.c:1232)
==28543== by 0x692973: Py_InitializeFromConfig (pylifecycle.c:1263)
==28543== by 0x41D9F9: init_from_config_clear (_testembed.c:68)
==28543== by 0x41DA9E: _testembed_Py_Initialize (_testembed.c:81)
==28543== by 0x41DC70: test_repeated_init_exec (_testembed.c:173)
==28543== by 0x421754: main (_testembed.c:2124)
==28543==
(...)
==28543==
==28543== LEAK SUMMARY:
==28543== definitely lost: 0 bytes in 0 blocks
==28543== indirectly lost: 0 bytes in 0 blocks
==28543== possibly lost: 1,056 bytes in 6 blocks
==28543== still reachable: 0 bytes in 0 blocks
==28543== suppressed: 0 bytes in 0 blocksBut in fact, this "leak" are just global variables (_Py_path_config) initialized once and not cleared explicitly. Py_RunMain() calls pymain_free() and so clear these globals, but Py_Initialize()+Py_Finalize() doesn't.
vstinner
commented
1 year ago On Windows, memory usage of PCbuild\amd64\_testembed_d.exe test_repeated_init_exec pass according to Task Manager, at each iteration:
It doesn't look like a memory leak to me: the memory usage looks stable.
Note: I edited Programs/_testembed.c to have 40 iterations.
ncoghlan
commented
1 year ago @vstinner _testembed.c doesn't call Py_InitializeEx, it calls its own init helper which correctly calls PyConfig_Clear. Py_InitializeEx appears to be missing that expected PyConfig_Clear call, so it may be triggering a leak report that _testembed wouldn't currently pick up.
vstinner
commented
1 year ago Oh, I forgot that I rewrote _testembed_Py_Initialize() to avoid the deprecated Py_Initialize() function :-) The function name is now misleading.
vstinner
commented
1 year ago _testembed_Py_Initialize() calls:
_PyConfig_InitCompatConfig(&config);
PyConfig_SetString(config, &config->program_name, L"./_testembed");
Py_InitializeFromConfig(config);Py_Initialize() is a thin-wrapper to _PyConfig_InitCompatConfig() + Py_InitializeFromConfig():
void Py_Initialize(void) { Py_InitializeEx(1); }
void
Py_InitializeEx(int install_sigs)
{
...
PyConfig config;
_PyConfig_InitCompatConfig(&config);
config.install_signal_handlers = install_sigs;
status = Py_InitializeFromConfig(&config);
...
}Just in case, I did another test with Py_Initialize():
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d635c5a4ab..90550388a3 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -31,7 +31,7 @@ char **main_argv;
/* Use path starting with "./" avoids a search along the PATH */
#define PROGRAM_NAME L"./_testembed"
-#define INIT_LOOPS 4
+#define INIT_LOOPS 40
// Ignore Py_DEPRECATED() compiler warnings: deprecated functions are
// tested on purpose here.
@@ -168,13 +168,18 @@ static int test_repeated_init_exec(void)
fprintf(stderr, "--- Loop #%d ---\n", i);
fflush(stderr);
- _testembed_Py_Initialize();
+ Py_SetProgramName(L"./_testembed");
+ Py_Initialize();
int err = PyRun_SimpleString(code);
Py_Finalize();
if (err) {
return 1;
}
+
+ printf("press a key to continue\n");
+ getchar();
}
+
return 0;
}
PCbuild\amd64\_testembed_d.exe test_repeated_init_exec pass memory usage in Task Manager:
For me, that's not a leak. For example, if they would be a constant leak of 0.1 MB per iteration, the memory would increase from 2.7 MB to 6.7 MB. But here it's more or less stable, between 3.1 and 3.5 MB.
IMO there is no leak and I suggest to just close the issue.
AlexSoft73
commented
1 year ago Hi to all and Thanks for the answers, Sorry I am a bit late I was out of my work bench and while I was able to read thru emails this conversation I had no source files, nor test enviroment with me.
The platform where I plan to use python is Windows, I cannot speak for Linux/OS/Unix I am not working currently on those platforms
Now, I forked the Python project and recompiled it using msvs2019 and I made a simple modification to 3 functions in obmalloc.c file:
The functions I mdified were:
-_static void _PyMem_RawMalloc(void Py_UNUSED(ctx), size_t size) (Increment counter) -static void _PyMem_RawCalloc(void Py_UNUSED(ctx), size_t nelem, size_t elsize) (Increment counter) -static void _PyMem_RawFree(void Py_UNUSED(ctx), void ptr) (Decrement counter)
when I finished the application, the counters math should be 0, but the result is 14, this means to me that there were 14 calls to "malloc", "calloc" that never got its counter part "free".
MallocCallocCounter: 1123 FreeCounter:1109
We can close this issue if this is the common desire.
Thanks is n advance Alexei
AlexSoft73
commented
1 year ago Hi ncoghlan
Do you use Windows or Unix based OS?
Thanks for your time. Alexei
AlexSoft73
commented
1 year ago Hi vstinner and thanks for your replay
You might not consider a leak going from Loop 1: 2.7 MB to Loop 40: 3.5 MB. I do see a leak there,
Stress up your test and take up to 100000 loops to see about the that number.
Thanks a lot for your time, Alexei
vstinner
commented
1 year ago I close the issue. This is no memory leak. It's not perfect, but it's good enough for now :-) Python 3.11 is way better than Python 3.10 and older versions regarding memory leaks when Python is embedded in an application.
You might not consider a leak going from Loop 1: 2.7 MB to Loop 40: 3.5 MB.
You're right: I don't consider it as a leak. Read again what I wrote. If you have a clear reproducer, please open a new issue.
A clear reproducer would mean that each Py_Initialize+Py_Finalize would leak a fixed amount of memory LEAK bytes, and that repeat it N times would leak LEAK x N bytes. It's not the case here.
when I finished the application, the counters math should be 0, but the result is 14, this means to me that there were 14 calls to "malloc", "calloc" that never got its counter part "free".
See my comment about pymain_free(): right, Py_Finalize() doesn't clear all global variables. There are a few bytes which remain allocated, but that's very small (1 kB), and this memory is not growing at each iteration. If it's already allocated, Python just uses it at the next iteration. It's not perfect, but it's acceptable for now.
If someone can propose a design to release this memory with Py_Initialize+Py_Finalize API, I'm interested.
AlexSoft73
commented
1 year ago But, if you close the issue how are you going to see it. I thought this exchange was to try and fix/improve Python code.
Thanks any ways,
vstinner
commented
1 year ago If someone has a reproducer as I described, please open a new issue.
If someone has an idea to fix the small leak of 1 kB, please open a new issue.
But the bug described in the issue is not a bug: there is no "leak".
AlexSoft73
commented
1 year ago Are you tesdting this is Windows or Unix derived system?
Thanks again,
Alexei
vstinner
commented
1 year ago Are you tesdting this is Windows or Unix derived system?
My comments where I wrote "On Windows", "Task Manager", "_testembed.exe", etc. are on Windows.
AlexSoft73
commented
1 year ago Ok vstinner
if you think python has no leaks specially, the memory allocated for globals which makes the Python dll not thread safe, and still don't believe the email where I detected that there are 14 calls requesting memory which memory is never released, that tells me, you use Python as a standalone application, hence never going to see this issue.
Any ways, thanks Alexei
ncoghlan
commented
1 year ago I'm still investigating this, as https://github.com/python/cpython/commit/6415e2ee4955b1a995c1e75544e2506b03780c3d is a 3.11 specific change that dropped the embedding tests' coverage of Py_InitializeEx. Prior to Python 3.11 a missing PyConfig_Clear call would have been entirely lost in the noise of all of the other process level allocations that Py_Finalize wasn't able to clear up.
While I could just add that apparently missing call (along with a new embedding test to specifically cover the convenience/compatibility APIs) and call it done, I'm first seeing if I can find a diagnostic that can clearly track whether the strings in the config struct (which are allocated with PyMem_RawMalloc) are being cleaned up correctly. (The config struct has 21 wide string fields and 5 lists of wide strings that may or may not be populated based on how the interpreter is being initialised, so a missing PyConfig_Clear could readily account for the 14 missing calls to PyMem_RawFree that @AlexSoft73 is detecting).
The refcount tracking and allocated block tracking don't pick those up since they only cover pymalloc and actual Python objects, so I'll be checking valgrind next. If that still doesn't clearly demonstrate a difference between Py_InitializeFromConfig (with a PyConfig_Clear call) and the Py_Initialize wrapper then I'll also resort to the global raw allocation counter approach)
ncoghlan
commented
1 year ago OK, the valgrind results are identical regardless of whether PyConfig_Clear is called or not, and the reason relates to the way that Py_InitializeEx uses PyConfig: it leaves all the dynamically allocated fields empty, so only the interpreter's runtime config struct gets those populated, the passed in configuration doesn't. Calling PyConfig_Clear anyway is still the more correct thing to do, but omitting it isn't actually going to leak anything if the API client never calls PyConfig_Read or one of the other PyConfig_ APIs that populates a dynamically allocated config field.
There are a handful of unmatched PyMem_RawMalloc calls (related to _Py_path_config as @vstinner noted above), but those don't get repeated (later interpreter instances reuse the previous allocations) so the total memory use doesn't grow over time.
What you do get over repeated invocations is increased memory fragmentation in the overall process, which can result in the kind of memory creep reported here without being a true leak.
Hi dear Python team,
I am a building a c/c++ application for general purpose and I would like to have in the application the possibility of running “Python” scripting code. Python has proved to be a good programming language to deal with heavy Math as in AI.
Now when I embed Python in my application things go perfect, except for memory leaks! As a sample by just issuing the following code leaves memory leaks of around (2Mb):
define PY_SSIZE_T_CLEAN
define Py_LIMITED_API 0x03000000
include
pragma comment (lib,"python3.lib")
int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow) { Py_InitializeEx(0); //For Embeded Python Py_FinalizeEx(); ..More code.. return 0; } Is there something I must do to cause Python to release all resources allocated