Memory leaks when embedded interpreter is reinitialized

[ab6d80f7-e9d3-4861-aa0c-6149f14e886b]

BPO	21387
Nosy	@ncoghlan, @pitrou, @vstinner
Superseder	bpo-1635741: Py_Finalize() doesn't clear all Python objects at exit

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['extension-modules', 'interpreter-core'] title = 'Memory leaks when embedded interpreter is reinitialized' updated_at = user = 'https://bugs.python.org/EvgeniyStepanov' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Extension Modules', 'Interpreter Core'] creation = creator = 'Evgeniy.Stepanov' dependencies = [] files = [] hgrepos = [] issue_num = 21387 keywords = [] message_count = 7.0 messages = ['217513', '217535', '217540', '217541', '218517', '236023', '355192'] nosy_count = 4.0 nosy_names = ['ncoghlan', 'pitrou', 'vstinner', 'Evgeniy.Stepanov'] pr_nums = [] priority = 'normal' resolution = 'duplicate' stage = 'resolved' status = 'closed' superseder = '1635741' type = None url = 'https://bugs.python.org/issue21387' versions = ['Python 2.7'] ```

[ab6d80f7-e9d3-4861-aa0c-6149f14e886b]

Following https://docs.python.org/2/c-api/init.html#Py_Finalize, I'm reinitializing embedded python interpreter multiple time in one process.

#include <Python.h>

void f() {
  Py_Initialize();
  PyRun_SimpleString("from time import time,ctime\n"
      "import datetime\n"
      "print 'Today is',ctime(time())\n");
  Py_Finalize();
}

int main(void) {
  for (int i = 0; i < 30; ++i)
    f();
  return 0;
}

I see 2 sources of memory leaks:

• _once_registry is never teared down.

==20144==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8120 byte(s) in 29 object(s) allocated from:

0 0x4a3e38 in malloc /code/llvm/build0/../projects/compiler-rt/lib/asan/asan_malloc_linux.cc:75

#1 0x4e1524 in \_PyObject_GC_Malloc build/Python-2.7.6/Modules/gcmodule.c:1499
#2 0x4e16ff in \_PyObject_GC_New build/Python-2.7.6/Modules/gcmodule.c:1521
#3 0x57e581 in PyDict_New build/Python-2.7.6/Objects/dictobject.c:277
#4 0x62b9a7 in \_PyWarnings_Init build/Python-2.7.6/Python/_warnings.c:898
#5 0x4c2985 in Py_InitializeEx build/Python-2.7.6/Python/pythonrun.c:254
#6 0x4c22a4 in f() build/Python-2.7.6/1.cc:4
#7 0x4c22a4 in main build/Python-2.7.6/1.cc:14

• datetime module creates a bunch of objects in its initialization function and never destroys them

Direct leak of 8120 byte(s) in 29 object(s) allocated from:

0 0x4a3e38 in malloc /code/llvm/build0/../projects/compiler-rt/lib/asan/asan_malloc_linux.cc:75

#1 0x4e1524 in \_PyObject_GC_Malloc build/Python-2.7.6/Modules/gcmodule.c:1499
#2 0x4e16ff in \_PyObject_GC_New build/Python-2.7.6/Modules/gcmodule.c:1521
#3 0x57e581 in PyDict_New build/Python-2.7.6/Objects/dictobject.c:277
#4 0x6cb976 in PyErr_NewException build/Python-2.7.6/Python/errors.c:577
#5 0x757227 in initzipimport build/Python-2.7.6/./Modules/zipimport.c:1235
#6 0x6e5797 in init_builtin build/Python-2.7.6/Python/import.c:1999
#7 0x6e158d in load_module build/Python-2.7.6/Python/import.c:1928
#8 0x6e6919 in import_submodule build/Python-2.7.6/Python/import.c:2700
#9 0x6e5ac7 in load_next build/Python-2.7.6/Python/import.c:2515
#10 0x6e036c in import_module_level build/Python-2.7.6/Python/import.c:2224
#11 0x6e036c in PyImport_ImportModuleLevel build/Python-2.7.6/Python/import.c:2288
#12 0x671e6f in builtin___import__ build/Python-2.7.6/Python/bltinmodule.c:49
#13 0x502b40 in PyObject_Call build/Python-2.7.6/Objects/abstract.c:2529
#14 0x502e82 in call_function_tail build/Python-2.7.6/Objects/abstract.c:2561
#15 0x502e82 in PyObject_CallFunction build/Python-2.7.6/Objects/abstract.c:2585
#16 0x6df7b0 in PyImport_Import build/Python-2.7.6/Python/import.c:2886
#17 0x6db9a7 in PyImport_ImportModule build/Python-2.7.6/Python/import.c:2129
#18 0x6db9a7 in \_PyImportHooks_Init build/Python-2.7.6/Python/import.c:239
#19 0x4c287f in Py_InitializeEx build/Python-2.7.6/Python/pythonrun.c:248
#20 0x4c22a4 in f() build/Python-2.7.6/1.cc:4
#21 0x4c22a4 in main build/Python-2.7.6/1.cc:14

[pitrou]

Thanks. It would be nice if you could try the same with Python 3.4, or the development version.

[5531d0d8-2a9c-46ba-8b8b-ef76132a492c]

Are we fixing these on a case by case basis or is it hopeless (msg146615)?

[pitrou]

I think fixing on a case by case is fine.

[5531d0d8-2a9c-46ba-8b8b-ef76132a492c]

I've run Evgeniy's example under Valgrind and changed it to import various C extensions. Unfortunately they all leak more or less, so perhaps we can revisit this when (if?) the PEP-3121 and PEP-384 changes have been implemented.

[ncoghlan]

For the record, the open issues about applying 3121 and 384 to the standard libary: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3

[vstinner]

Duplicate of bpo-1635741.