search

python / cpython

The Python programming language
https://www.python.org
Other
62.32k stars 29.94k forks source link

`Py_Initialize()` still leaks in libpython3.11 #100773

Open Gerzer opened 1 year ago

Gerzer commented 1 year ago

Bug Report

The following code leaks several hundred kilobytes with the latest libpython from the Ubuntu APT repository:

#include <Python.h>

int main(void) {
        Py_Initialize();
        Py_Finalize();
        return 0;
}

With libpython3.10, it consistently leaks 480 KB, whereas with libpython3.11, it consistently leaks 408 KB. (These are proper IEC power-of-10 kilobytes, not power-of-2 kibibytes.) I’ve verified this by running this code as a Clang-compiled dynamic executable through Valgrind.

Binary Images

libpython3.10 version:

linux-vdso.so.1 (0x0000ffffbf695000)
libpython3.10.so.1.0 => /lib/aarch64-linux-gnu/libpython3.10.so.1.0 (0x0000ffffbf050000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffbeea0000)
libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffffbee60000)
libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffffbee30000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffbed90000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffbf65c000)

libpython3.11 version:

linux-vdso.so.1 (0x0000ffffaee59000)
libpython3.11.so.1.0 => /lib/aarch64-linux-gnu/libpython3.11.so.1.0 (0x0000ffffae650000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffae4a0000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffae400000)
libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffffae3d0000)
libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffffae390000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffaee20000)

Valgrind Reports

Environment

sobolevn commented 1 year ago

cc @vstinner who worked on removing the startup leakages as far as I remember :)

doctorlai-msrc commented 1 year ago

Leaking detected by Address Santitizer as well:

=================================================================
==22370==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 267212 byte(s) in 100 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d9e7  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a9e7)

Direct leak of 536 byte(s) in 1 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d182  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a182)

Indirect leak of 25922 byte(s) in 26 object(s) allocated from:
    #0 0x7f0a1c916808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x7f0a1c50d9e7  (/lib/x86_64-linux-gnu/libpython3.8.so.1.0+0x25a9e7)

SUMMARY: AddressSanitizer: 293670 byte(s) leaked in 127 allocation(s).
vstinner commented 1 year ago

It's different if memory is allocated exactly once and never released explicitly by Py_Finalize(), or if each Py_Initialize() call allocates new memory. Most of remaining "leaks" are memory allocated exactly once and then never released by Py_Finalize().

There is still a work-in-progress to convert remaining stdlib extensions to multi-phase initialization. A recent example with _elementtree: https://github.com/python/cpython/commit/fee7a995a18589001a432cea365fd04bf8efb7c1

If you are curious, you should dig into the pymain_free() function of Modules/main.c to see memory not freed by Py_Finalize(), but freed by Py_RunMain().

Mekk commented 9 months ago

Python 3.12 arrived with plenty of new sanitizer-reported errors :-( ( https://github.com/python/cpython/issues/113190 )