Convert static types to heap types: use PyType_FromSpec()

[corona10]

BPO	40077
Nosy	@pitrou, @vstinner, @phsilva, @encukou, @skrah, @ericsnowcurrently, @serhiy-storchaka, @corona10, @pablogsal, @miss-islington, @shihai1991, @erlend-aasland, @h-vetinari, @nw0, @kumaraditya303, @CharlieZhao95
PRs	python/cpython#19177 python/cpython#19202 python/cpython#19341 python/cpython#19344 python/cpython#19438 python/cpython#20960 python/cpython#20974 python/cpython#21953 python/cpython#21954 python/cpython#23108 python/cpython#23124 python/cpython#23136 python/cpython#23428 python/cpython#23443 python/cpython#23975 python/cpython#24065 python/cpython#24066 python/cpython#24481 python/cpython#30884
Files	bench_isinstance_check.py bench_subclass_check.py bench_isinstance_check.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'type-feature', '3.10'] title = 'Convert static types to heap types: use PyType_FromSpec()' updated_at = user = 'https://github.com/corona10' ``` bugs.python.org fields: ```python activity = actor = 'kumaraditya' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules'] creation = creator = 'corona10' dependencies = [] files = ['49007', '49008', '49009'] hgrepos = [] issue_num = 40077 keywords = ['patch'] message_count = 47.0 messages = ['365087', '365088', '365117', '365125', '365142', '365145', '365149', '365154', '365205', '365220', '365241', '365242', '365255', '365257', '365317', '365558', '365773', '366024', '366061', '368669', '371120', '371813', '371890', '372073', '372074', '372076', '372326', '375943', '380109', '380267', '380516', '380517', '380518', '383848', '383908', '384228', '384236', '384271', '384272', '384293', '410810', '411630', '411903', '411904', '411907', '411964', '412338'] nosy_count = 16.0 nosy_names = ['pitrou', 'vstinner', 'phsilva', 'petr.viktorin', 'skrah', 'eric.snow', 'serhiy.storchaka', 'corona10', 'pablogsal', 'miss-islington', 'shihai1991', 'erlendaasland', 'h-vetinari', 'nw0', 'kumaraditya', 'CharlieZhao'] pr_nums = ['19177', '19202', '19341', '19344', '19438', '20960', '20974', '21953', '21954', '23108', '23124', '23136', '23428', '23443', '23975', '24065', '24066', '24481', '30884'] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue40077' versions = ['Python 3.10'] ```

[corona10]

Some of modules is not using PyType_FromSpec. We need to convert them.

This changes can bring

• allow to destroy types at exit!
• allow subinterpreters to have their own "isolated" typ

[vstinner]

We need to convert them.

Let me elaborate. Static types have multiple issues:

• Their lifetime is not well defined.
• It is not obvious when they are ready to be used.
• They are not destroyed at exit.
• They are incompatible with subinterpreters: each interpreter should have its own copy of a type, rather than static types are shared by all interpreters which cause problems with reference counting (require GIL or atomic operation).
• They are causing issues with stable ABI (limited C API): PEP-384.

[pablogsal]

Wouldn't having less static types slow down startup time?

[vstinner]

Pablo:

Wouldn't having less static types slow down startup time?

That's possible, even if I only expect a minor or non significant overhead.

But before starting talking about performances, we should focus on the correctness. Static types are causing issues with subinterpreters and the Python finalization.

[vstinner]

New changeset 33f15a16d40cb8010a8c758952cbf88d7912ee2d by Dong-hee Na in branch 'master': bpo-40077: Convert _json module to use PyType_FromSpec() (GH-19177) https://github.com/python/cpython/commit/33f15a16d40cb8010a8c758952cbf88d7912ee2d

[5531d0d8-2a9c-46ba-8b8b-ef76132a492c]

Wouldn't having less static types slow down startup time?

Yes, and not only startup time:

https://bugs.python.org/issue15722

[vstinner]

In the _json module, PyModule_GetState() (called by get_json_state()) is only used by the garbage collector (traverse function) and to unload the module (clear and free functions). It's not used in "hot code" (let me consider that the GC is not part of the usual "hot code" :-)).

But maybe we should measure the overhead of future changes if PyModule_GetState() starts to be used in "hot code" by running a few microbenchmarks.

--

Stefan Krah:

Yes, and not only startup time: https://bugs.python.org/issue15722

Aha, that's interesting. I didn't know that it could have an impact on runtime performance as well.

_decimal_pep3121-384_v1.patch attached to bpo-15722 seems to use:

#define _decimal_state_global ((_decimal_state *)PyModule_GetState(PyState_FindModule(&_decimal_module)))

whereas the commit 33f15a16d40cb8010a8c758952cbf88d7912ee2d only uses:

static inline _jsonmodulestate*
get_json_state(PyObject *module)
{
    void *state = PyModule_GetState(module);
    assert(state != NULL);
    return (_jsonmodulestate *)state;
}

Maybe PyState_FindModule() adds a little overhead, even if this function is simple: in short, it gets the i-th item of a list (from PyInterpreterState.modules_by_index).

PyModule_GetState() function is really simple: it only reads PyModuleObject.md_state attribute.

Or maybe _decimal_state_global was used in "hot code".

If PyState_FindModule() or PyModule_GetState() is the source of the overhead, maybe we could try to optimise these functions, or pass directly the module state to inner functions.

--

PyState_FindModule() doesn't work for a module using PyModuleDef_Init(). The PEP-573 is going to give access to the module state in functions which didn't access to it previsouly.

The commit 33f15a16d40cb8010a8c758952cbf88d7912ee2d removed a few assertions checking that "self" has the expected type. It wasn't possible to get the module state to get the types, because PEP-573 is not implemented yet and PyState_FindModule() doesn't work in _json (which uses PyModuleDef_Init()). I decided that it's ok to remove these assertions: it should not be possible to call these functions with another type in practice.

--

In his PR 19177, Dong-hee started by replacing direct access to PyTypeObject fields, like replacing:

type->free(self);

with:

freefunc free_func = PyType_GetSlot(tp, Py_tp_free);
free_func(self);

I asked him to revert these changes.

I'm interested to experiment a few C extension modules of the stdlib using the limited C API (PEP-384, stable ABI), but I don't think that it should done while converting modules to PyType_FromSpec().

We should separate the two changes. And I would prefer to first see the overhead of PyType_FromSpec(), and discuss the advantages and drawbacks.

[5531d0d8-2a9c-46ba-8b8b-ef76132a492c]

Or maybe _decimal_state_global was used in "hot code".

Yes, _decimal has problems here that most modules don't have. Modules like atexit are obviously fine. :)

I just posted it as an example why one should be a bit cautious.

The PEP-573 is going to give access to the module state in functions which didn't access to it previously.

That could help, we'll see.

[corona10]

And I would prefer to first see the overhead of PyType_FromSpec(), and discuss the advantages and drawbacks.

Should we stop the work until the overhead is measured?

[vstinner]

Should we stop the work until the overhead is measured?

Can you try to measure the _abc._abc_instancecheck() and _abc._abc_subclasscheck() functions performance before/after your change? Functions like register() are rarely call, so I don't care much of their performance (I expect a small overhead or no overhead).

[corona10]

It shown 1% slower for performance.

+--------------------------+-----------------+----------------------------+ | Benchmark | master-subclass | bpo-40077-subclass | +==========================+=================+============================+ | bench _abc_subclasscheck | 295 ns | 300 ns: 1.01x slower (+1%) | +--------------------------+-----------------+----------------------------+

+--------------------------+-------------------+----------------------------+ | Benchmark | master-isinstance | bpo-40077-isinstance | +==========================+===================+============================+ | bench _abc_instancecheck | 229 ns | 232 ns: 1.01x slower (+1%) | +--------------------------+-------------------+----------------------------+

[corona10]

Can you try to measure the _abc._abc_instancecheck() and _abc._abc_subclasscheck()

I 've submitted the benchmark :)

[vstinner]

IMO 1.01x slower on a *microbenchmark* is not significant so it's ok. Thanks for checking.

You pass a *type to isinstance() in bench_isinstance_check.py. You should pass *an instance* instead. Like (complete the (...) ;-)):

runner.timeit(name="bench _abc_instancecheck", stmt="isinstance(obj, T)", setup = """from abc import ABC (...) obj = (1, 2, 3)""")

Would you mind to fix your microbenchmark and re-run it?

[corona10]

You pass a *type to isinstance() in bench_isinstance_check.py.

Thanks for the catch my mistake.

The result is showing: Not significant (1): bench _abc_instancecheck

[vstinner]

New changeset 53e4c91725083975598350877e2ed8e2d0194114 by Dong-hee Na in branch 'master': bpo-40077: Convert _abc module to use PyType_FromSpec() (GH-19202) https://github.com/python/cpython/commit/53e4c91725083975598350877e2ed8e2d0194114

[vstinner]

bpo-40077: Convert _abc module to use PyType_FromSpec() (GH-19202)

This change introduced a reference leak: bpo-40149 "test_threading leaked [38, 38, 38] references, sum=114".

[vstinner]

New changeset b709302f3125622986bd458dfb2954fda5e8366d by Hai Shi in branch 'master': bpo-40077: Fix potential refleaks of _json: traverse memo (GH-19344) https://github.com/python/cpython/commit/b709302f3125622986bd458dfb2954fda5e8366d

[ericsnowcurrently]

Wouldn't having less static types slow down startup time?

FWIW, I've been considering an approach where the main interpreter keeps using static types but subinterpreters use heap types. If it isn't too much effort (or too hacky) then it might be a sufficient solution for now.

[corona10]

New changeset dcb04d9c6dd6f31449ade6765fa4d26a305e7381 by Hai Shi in branch 'master': bpo-40077: Remove redundant cast in json module (GH-19438) https://github.com/python/cpython/commit/dcb04d9c6dd6f31449ade6765fa4d26a305e7381

[vstinner]

See also bpo-40601: [C API] Hide static types from the limited C API.

[vstinner]

I tried to finalize static types in Py_Finalize(), but it didn't work:

• https://bugs.python.org/issue1635741#msg371119
• https://github.com/python/cpython/pull/20763

[vstinner]

PR 20960 (_bz2 module) triggers an interesting question. The effect of converting a static type to a heap type on pickle, especially for protocols 0 and 1.

pickle.dumps(o, 0) calls object.__reduce__(o) which calls copyreg._reduce_ex(o, 0).

copyreg._reduce_ex() behaves differently on heap types:

...
for base in cls.__mro__:
    if hasattr(base, '__flags__') and not base.__flags__ & _HEAPTYPE:
        break
else:
    base = object # not really reachable
...

There are 3 things which impacts serialization/deserialization:

• Py_TPFLAGS_HEAPTYPE flag in the type flags
• Is __new__() overriden in the type?
• Py_TPFLAGS_BASETYPE flag in the type flags

Examples:

• In Python 3.7, _random.Random() cannot be serialized because it's a static type (it doesn't have (Py_TPFLAGS_HEAPTYPE)
• In master, _random.Random() cannot be deserialized because _random.Random() has __new() method (it's not object.__new())

[corona10]

New changeset ec689187957cc80af56b9a63251bbc295bafd781 by Dong-hee Na in branch 'master': bpo-40077: Convert _bz2 module to use PyType_FromSpec (GH-20960) https://github.com/python/cpython/commit/ec689187957cc80af56b9a63251bbc295bafd781

[vstinner]

Search also for issues with "384" in their title (34 open issues):

https://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=384&%40columns=title&id=&%40columns=id&stage=&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=&priority=&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40group=&%40pagesize=50&%40startwith=0&%40sortdir=on&%40queryname=&%40old-queryname=&%40action=search

Search for issues with PEP-3121 keyword (40 open issues):

https://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=&versions=&dependencies=&assignee=&keywords=13&priority=&status=1&%40columns=status&resolution=&nosy_count=&message_count=&%40group=&%40pagesize=50&%40startwith=0&%40sortdir=on&%40queryname=&%40old-queryname=&%40action=search

[vstinner]

For example, see bpo-15849 for the xx module.

[vstinner]

See meta bpo-15787 "PEP-3121, 384 Refactoring" which tracks all these issues as dependencies.

[vstinner]

The Tools/scripts/abitype.py script can help to port C extensions to PyType_FromSpec().

[corona10]

New changeset 31967fd8d0220af84e2698172d1378bffc8cd851 by Dong-hee Na in branch 'master': bpo-40077: Convert _operator to use PyType_FromSpec (GH-21954) https://github.com/python/cpython/commit/31967fd8d0220af84e2698172d1378bffc8cd851

[vstinner]

Count static types:

grep -E 'static PyTypeObject . =' $(find -name "\.c"|grep -v Doc/)|wc -l

Count heap types:

grep -E 'PyType_Spec . =' $(find -name "\.c")|wc -l

Status:

• 3.6: 10 heap (6%) vs 145 static (total: 155)
• 3.7: 10 heap (6%) vs 150 static (total: 160)
• 3.8: 15 heap (9%) vs 157 static (total: 172)
• 3.9: 39 heap (21%) vs 149 static (total: 188) -- 2.6x more heap types than 3.8!
• master: 69 heap (35%) vs 131 static (total: 200) -- 1.8x more heap types than 3.9!

The percentage of heap static is growing in the right direction ;-) The total number of types is also growing at each Python realease!

[vstinner]

New changeset 74b4eda98b650245216292ace8d7e30d3997baa7 by Erlend Egeberg Aasland in branch 'master': bpo-40077: Convert mmap.mmap static type to a heap type (GH-23108) https://github.com/python/cpython/commit/74b4eda98b650245216292ace8d7e30d3997baa7

[miss-islington]

New changeset 01c6aa43dc56b3b64d584c58a49c86f816c05a91 by Erlend Egeberg Aasland in branch 'master': bpo-40077: Convert _queuemodule to use heap types (GH-23136) https://github.com/python/cpython/commit/01c6aa43dc56b3b64d584c58a49c86f816c05a91

[pitrou]

Thank you @erlendaasland !

[pitrou]

Oops, sorry, I hadn't noticed this was a catch-all issue. Reopening.

[vstinner]

Progress: 43% (89/206) of types are declared as heap types on a total of 206 types.

$ grep -E 'static PyTypeObject .* =' $(find -name "*.c"|grep -v Doc/)|wc -l
117
$ grep -E 'PyType_Spec .* =' $(find -name "*.c")|wc -l
89

[miss-islington]

New changeset bf108bb21e1d75e30bd17141cc531dd08a5e5d0c by Erlend Egeberg Aasland in branch 'master': bpo-40077: Fix typo in simplequeue_get_state_by_type() (GH-23975) https://github.com/python/cpython/commit/bf108bb21e1d75e30bd17141cc531dd08a5e5d0c

[encukou]

New changeset 75bf107c62fbdc00af51ee4f6ab69df4bd201104 by Erlend Egeberg Aasland in branch 'master': bpo-40077: Convert arraymodule to use heap types and establish module state (GH-23124) https://github.com/python/cpython/commit/75bf107c62fbdc00af51ee4f6ab69df4bd201104

[erlend-aasland]

Should Modules/_testcapimodule.c stay untouched?

[encukou]

New changeset b8eb3765908c0063f0739595ba4b296cc8863d19 by Erlend Egeberg Aasland in branch 'master': bpo-40077: Add traverse/clear/free to arraymodule (GH-24066) https://github.com/python/cpython/commit/b8eb3765908c0063f0739595ba4b296cc8863d19

[encukou]

Yes, please keep _testcapimodule.c as it is. Static types are still supported and need to be tested.

[erlend-aasland]

Ok, perhaps we should leave a comment about that? It already has a comment about multi-phase init.

What about Modules/_testbuffer.c and the xx-modules?

$ grep -E 'static PyTypeObject .* =' $(find . -name "*.c"|grep -vE '(Doc/|Modules/_testcapimodule)') | wc -l
94

$ grep -E 'PyType_Spec .* =' $(find . -name "*.c")|wc -l                                                                                                                            (master)cpython.git
95

We're almost halfway there.

[vstinner]

See also bpo-46417 "[subinterpreters] Clear static types in Py_Finalize()".

[vstinner]

Converting further static types to heap types require a discussion. See what the Python Steering Council wrote at Feb 8, 2021:

"The Steering Council discussed the ongoing work on porting types in the standard library to heap-types and the subinterpreter-related changes. It was decided that through Pablo, the Steering Council will ask the core developers driving those changes to create an informational PEP and not to make any more changes in this area after beta 1, as per our general policy."

https://github.com/python/steering-council/blob/1d85eefdc5861a096c3859e9990dbc8527c5973b/updates/2021-02-steering-council-update.md

[vstinner]

I used the following shell command to search remaining static types: --- grep -E 'static PyTypeObject [a-zA-Z_0-9]+ *(;|= *{|) *$' $(find -name "*.c") ---

I found 86 static types in 17 files:

• PC/_msi.c:

  • msidb_Type
  • msiview_Type
  • record_Type
  • summary_Type

• Modules/_ctypes/_ctypes.c:

  • DictRemover_Type
  • PyComError_Type
  • PyDec_Type
  • Simple_Type
  • StructParam_Type
  • Struct_Type
  • UnionType_Type
  • Union_Type

• Modules/_decimal/_decimal.c:

  • PyDecContextManager_Type
  • PyDecContext_Type
  • PyDecSignalDict_Type

• Modules/xxmodule.c:

  • Null_Type
  • Str_Type
  • Xxo_Type

• Modules/_testbuffer.c:

  • NDArray_Type
  • StaticArray_Type

• Modules/itertoolsmodule.c (multiphase init):

  • _grouper_type
  • accumulate_type
  • combinations_type
  • compress_type
  • count_type
  • cwr_type
  • cycle_type
  • dropwhile_type
  • filterfalse_type
  • groupby_type
  • pairwise_type
  • permutations_type
  • starmap_type
  • takewhile_type
  • tee_type
  • teedataobject_type

• Modules/_xxsubinterpretersmodule.c:

  • ChannelIDtype

• Modules/_datetimemodule.c:

  • PyDateTime_DateTimeType
  • PyDateTime_DateType
  • PyDateTime_DeltaType
  • PyDateTime_IsoCalendarDateType
  • PyDateTime_TZInfoType
  • PyDateTime_TimeType
  • PyDateTime_TimeZoneType

• Modules/_testcapimodule.c:

  • ContainerNoGC_type
  • GenericAlias_Type
  • Generic_Type
  • MethClass_Type
  • MethInstance_Type
  • MethStatic_Type
  • MethodDescriptor2_Type
  • MethodDescriptorBase_Type
  • MethodDescriptorDerived_Type
  • MethodDescriptorNopGet_Type
  • MyList_Type
  • PyRecursingInfinitelyError_Type
  • _HashInheritanceTester_Type
  • awaitType
  • ipowType
  • matmulType
  • test_structmembersType

• Modules/_zoneinfo.c:

  • PyZoneInfo_ZoneInfoType

• Modules/ossaudiodev.c:

  • OSSAudioType
  • OSSMixerType

• Modules/socketmodule.c:

  • sock_type

• Modules/xxsubtype.c:

  • spamdict_type
  • spamlist_type

• Modules/_collectionsmodule.c:

  • defdict_type
  • deque_type
  • dequeiter_type
  • dequereviter_type
  • tuplegetter_type

• Modules/_elementtree.c:

  • ElementIter_Type
  • Element_Type
  • TreeBuilder_Type
  • XMLParser_Type

• Modules/_pickle.c:

  • Pdata_Type
  • PicklerMemoProxyType
  • Pickler_Type
  • UnpicklerMemoProxyType
  • Unpickler_Type

• Modules/_asynciomodule.c:

  • FutureIterType
  • FutureType
  • PyRunningLoopHolder_Type
  • TaskStepMethWrapper_Type
  • TaskType

[vstinner]

And I found 135 more static types with this command: --- grep -E '^PyTypeObject [a-zA-Z_0-9]+ *(;|= *{|) *$' $(find -name "*.c") ---

Types:

Objects/cellobject.c: PyCell_Type Objects/sliceobject.c: PyEllipsis_Type Objects/sliceobject.c: PySlice_Type Objects/unionobject.c: _PyUnion_Type Objects/methodobject.c: PyCFunction_Type Objects/methodobject.c: PyCMethod_Type Objects/picklebufobject.c: PyPickleBuffer_Type Objects/boolobject.c: PyBool_Type Objects/bytearrayobject.c: PyByteArray_Type Objects/bytearrayobject.c: PyByteArrayIter_Type Objects/classobject.c: PyMethod_Type Objects/classobject.c: PyInstanceMethod_Type Objects/fileobject.c: PyStdPrinter_Type Objects/genericaliasobject.c: Py_GenericAliasType Objects/interpreteridobject.c: _PyInterpreterID_Type Objects/iterobject.c: PySeqIter_Type Objects/iterobject.c: PyCallIter_Type Objects/iterobject.c: _PyAnextAwaitable_Type Objects/moduleobject.c: PyModuleDef_Type Objects/moduleobject.c: PyModule_Type Objects/rangeobject.c: PyRange_Type Objects/rangeobject.c: PyRangeIter_Type Objects/rangeobject.c: PyLongRangeIter_Type Objects/namespaceobject.c: _PyNamespace_Type Objects/bytesobject.c: PyBytes_Type Objects/bytesobject.c: PyBytesIter_Type Objects/capsule.c: PyCapsule_Type Objects/complexobject.c: PyComplex_Type Objects/dictobject.c: PyDict_Type Objects/dictobject.c: PyDictIterKey_Type Objects/dictobject.c: PyDictIterValue_Type Objects/dictobject.c: PyDictIterItem_Type Objects/dictobject.c: PyDictRevIterKey_Type Objects/dictobject.c: PyDictRevIterItem_Type Objects/dictobject.c: PyDictRevIterValue_Type Objects/dictobject.c: PyDictKeys_Type Objects/dictobject.c: PyDictItems_Type Objects/dictobject.c: PyDictValues_Type Objects/floatobject.c: PyFloat_Type Objects/frameobject.c: PyFrame_Type Objects/funcobject.c: PyFunction_Type Objects/funcobject.c: PyClassMethod_Type Objects/funcobject.c: PyStaticMethod_Type Objects/memoryobject.c: _PyManagedBuffer_Type Objects/memoryobject.c: PyMemoryView_Type Objects/odictobject.c: PyODict_Type Objects/odictobject.c: PyODictIter_Type Objects/odictobject.c: PyODictKeys_Type Objects/odictobject.c: PyODictItems_Type Objects/odictobject.c: PyODictValues_Type Objects/setobject.c: PySetIter_Type Objects/setobject.c: PySet_Type Objects/setobject.c: PyFrozenSet_Type Objects/setobject.c: _PySetDummy_Type Objects/tupleobject.c: PyTuple_Type Objects/tupleobject.c: PyTupleIter_Type Objects/object.c: _PyNone_Type Objects/object.c: _PyNotImplemented_Type Objects/unicodeobject.c: PyUnicode_Type Objects/unicodeobject.c: PyUnicodeIter_Type Objects/listobject.c: PyList_Type Objects/listobject.c: PyListIter_Type Objects/listobject.c: PyListRevIter_Type Objects/genobject.c: PyGen_Type Objects/genobject.c: PyCoro_Type Objects/genobject.c: _PyCoroWrapper_Type Objects/genobject.c: PyAsyncGen_Type Objects/genobject.c: _PyAsyncGenASend_Type Objects/genobject.c: _PyAsyncGenWrappedValue_Type Objects/genobject.c: _PyAsyncGenAThrow_Type Objects/longobject.c: PyLong_Type Objects/descrobject.c: PyMethodDescr_Type Objects/descrobject.c: PyClassMethodDescr_Type Objects/descrobject.c: PyMemberDescr_Type Objects/descrobject.c: PyGetSetDescr_Type Objects/descrobject.c: PyWrapperDescr_Type Objects/descrobject.c: _PyMethodWrapper_Type Objects/descrobject.c: PyDictProxy_Type Objects/descrobject.c: PyProperty_Type Objects/enumobject.c: PyEnum_Type Objects/enumobject.c: PyReversed_Type Objects/codeobject.c: _LineIterator Objects/codeobject.c: _PositionsIterator Objects/codeobject.c: PyCode_Type Objects/typeobject.c: PyType_Type Objects/typeobject.c: PyBaseObject_Type Objects/typeobject.c: PySuper_Type Python/bltinmodule.c: PyFilter_Type Python/bltinmodule.c: PyMap_Type Python/bltinmodule.c: PyZip_Type Python/context.c: PyContext_Type Python/context.c: PyContextVar_Type Python/context.c: PyContextToken_Type Python/context.c: _PyContextTokenMissing_Type Python/hamt.c: _PyHamtItems_Type Python/hamt.c: _PyHamtKeys_Type Python/hamt.c: _PyHamtValues_Type Python/hamt.c: _PyHamt_Type Python/hamt.c: _PyHamt_ArrayNode_Type Python/hamt.c: _PyHamt_BitmapNode_Type Python/hamt.c: _PyHamt_CollisionNode_Type Python/traceback.c: PyTraceBack_Type Python/symtable.c: PySTEntry_Type Modules/_ctypes/callproc.c: PyCArg_Type Modules/_ctypes/_ctypes.c: PyCStructType_Type Modules/_ctypes/_ctypes.c: PyCPointerType_Type Modules/_ctypes/_ctypes.c: PyCArrayType_Type Modules/_ctypes/_ctypes.c: PyCSimpleType_Type Modules/_ctypes/_ctypes.c: PyCFuncPtrType_Type Modules/_ctypes/_ctypes.c: PyCData_Type Modules/_ctypes/_ctypes.c: PyCFuncPtr_Type Modules/_ctypes/_ctypes.c: PyCArray_Type Modules/_ctypes/_ctypes.c: PyCPointer_Type Modules/_ctypes/cfield.c: PyCField_Type Modules/_ctypes/stgdict.c: PyCStgDict_Type Modules/_ctypes/callbacks.c: PyCThunk_Type Modules/_io/iobase.c: PyIOBase_Type Modules/_io/iobase.c: PyRawIOBase_Type Modules/_io/stringio.c: PyStringIO_Type Modules/_io/textio.c: PyTextIOBase_Type Modules/_io/textio.c: PyIncrementalNewlineDecoder_Type Modules/_io/textio.c: PyTextIOWrapper_Type Modules/_io/bytesio.c: PyBytesIO_Type Modules/_io/fileio.c: PyFileIO_Type; Modules/_io/fileio.c: PyFileIO_Type Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type; Modules/_io/winconsoleio.c: PyWindowsConsoleIO_Type Modules/_io/bufferedio.c: PyBufferedIOBase_Type Modules/_io/bufferedio.c: PyBufferedReader_Type Modules/_io/bufferedio.c: PyBufferedWriter_Type Modules/_io/bufferedio.c: PyBufferedRWPair_Type Modules/_io/bufferedio.c: PyBufferedRandom_Type Modules/_multiprocessing/semaphore.c: _PyMp_SemLockType Modules/_cursesmodule.c: PyCursesWindow_Type; Modules/_cursesmodule.c: PyCursesWindow_Type

[vstinner]

And I found 135 more static types with this command

Of these 135 static types, most are cleared since bpo-46417 was implemented:

• 103 types are cleared by _PyTypes_FiniTypes()
• 15 types are cleared by _PyIO_Fini()
• the remaining 17 types are not cleared at Python exit: types of _ctypes, _curses and _multiprocessing extensions

[vstinner]

I marked bpo-23769 "valgrind reports leaks for test_zipimport" as duplicate of this issue. At exit, Python doesn't clear the static types of the _collections, itertools and _struct extensions:

• itertools.accumulate
• itertools.combinations
• itertools.combinations_with_replacement
• itertools.cycle
• itertools.dropwhile
• itertools.takewhile
• itertools.islice
• itertools.starmap
• itertools.chain
• itertools.compress
• itertools.filterfalse
• itertools.count
• itertools.zip_longest
• itertools.pairwise
• itertools.permutations
• itertools.product
• itertools.repeat
• itertools.groupby
• itertools._grouper
• itertools._tee
• itertools._tee_dataobject
• collections.deque
• _collections._deque_iterator
• _collections._deque_reverse_iterator
• _collections._tuplegetter
• _struct.Struct
• _struct.unpack_iterator

See: https://bugs.python.org/issue23769#msg411963

[kumaraditya303]

On Windows PC/winreg.c has PyHKEY_Type static type which isn't cleared at exit too.

[kumaraditya303]

PEP 687 was accepted

[erlend-aasland]

PEP 687 was accepted

Note that this does not imply that all types shall be converted to heap types. This is also mentioned in the PEP; please read it carefully.

[kumaraditya303]

Note that this does not imply that all types shall be converted to heap types. This is also mentioned in the PEP; please read it carefully.

We are on the same page, every module needs to considered individually and I am against converting any built-in modules or types unless it's necessary.