Isolating modules in the stdlib

[encukou]

PEP 630 has motivations and technical notes. What needs to be documented better is how both applies to stdlib. Specifically:

• list the behavior changes when static types are converted to heap types (mutability, pickleability, any more?), and document how to undo them when the new behavior is not better. Same for any other behavior changes related to changing existing code.
• explain performance impacts, how to measure and analyze this.
• what to watch out for and test.

Whether that should be a new PEP or added to 630 doesn't matter much, IMO.

Then, any introduction of PEP 630 should be done deliberately, analyzing the pros and cons for each module separately. It should not be a mass change, and it should involve explaining the motivations & specific implications to module maintainers.

[erlend-aasland]

Thanks, Petr!

Relevant bpo's (I'll expand the list later):

• bpo-1635741: _PyFinalize() doesn't clear all Python objects at exit
• bpo-36876: Global C variables are a problem
• bpo-40077: _Convert static types to heap types: use PyTypeFromSpec()
• bpo-40137: TODO list when PEP 573 "Module State Access from C Extension Methods" will be implemented This bpo has interesting discussions regarding a specific performance degradation, and how that issue was resolved.
• bpo-41052: Opt out serialization/deserialization for heap type
• bpo-42972: _[C API] Heap types (PyTypeFromSpec) must fully implement the GC protocol Discussion regarding heap types and GC, and how to resolve these. Heap types must fully implement the GC protocol.
• bpo-43908: _array.array should remain immutable: add Py_TPFLAGSIMMUTABLETYPE flag Discussion regarding heap type mutability, and how this has been resolved by adding the Py_TPFLAGS_IMMUTABLETYPE flag.
• bpo-43916: _Mark static types newly converted to heap types as immutable: add Py_TPFLAGS_DISALLOWINSTANTIATION type flag (Original bpo-43916 subject: _Check that new heap types cannot be created uninitialised: add Py_TPFLAGS_DISALLOWINSTANTIATION type flag. I don't know why it was changed.) Discussion regarding heap type initialisation issues, and how they can be resolved using the Py_TPFLAGS_DISALLOW_INSTANTIATION type flag.
• bpo-45113: [subinterpreters][C API] Add a new function to create PyStructSequence from Heap

Related PEP's:

• PEP 3121: Extension Module Initialization and Finalization
• PEP 489: Multi-phase extension module initialization
• PEP 573: Extension Module Initialization and Finalization

[erlend-aasland]

• list the behavior changes when static types are converted to heap types (mutability, pickleability, any more?), and document how to undo them when the new behavior is not better. Same for any other behavior changes related to changing existing code.

Expanding this list a little bit (mostly as a note to self):

• Heap types are mutable by default, static types are immutable. We want to keep stdlib types immutable (see bpo-43908), so the Py_TPFLAGS_IMMUTABLETYPE flag must be used for all new heap types in the stdlib.
• Quoting Serhiy: _"Static type with tpnew = NULL does not have public constructor, but heap type inherits constructor from base class. As result, it allows to create instances without proper initialization, that can lead to crash." See bpo-43916. If your static type had tp_new = NULL, use the Py_TPFLAGS_DISALLOW_INSTANTIATION flag when converting it to a heap type.

[encukou]

• bpo-41052 talks about pickle issues.

[erlend-aasland]

Here's a draft of a ToC:

Isolating modules in the standard library

Subtitle: Applying PEP 630 to the standard library

• why are we doing this
  • recap rationale / motivation
  • list stdlib extension module benefits
  • bpo-1635741: Py_Finalize() doesn't clear all Python objects at exit
• heap types
  • howto
  • backwards compatibility checklist
  • mutability
  • pickling/serialization
  • default constructor / disallowing instantiation
  • gc protocol
• capsules
  • examples from the stdlib
• module state tips and tricks
  • short module state access intro (PEP 573) (Argument Clinic, new API's, etc.)?
  • storing state pointer in objects
  • passing state to callbacks (example from sqlite3 module?)
• performance
  • how to measure (pyperf, benchmark suite)
  • how to avoid performance degradation (functools example?)
• check-list for isolating extension modules in the standard library
  • open a discourse topic: talk to the code owner/maintainer/dev team (=> new discourse label?)
  • open a bpo ticket
  • identify performance bottlenecks involving module state
  • devise an implementation plan and a test strategy
• tutorial
  • walk through one or two examples from stdlib
  • refer to Modules/xxlimited.c

[encukou]

Applying PEP 630 to the standard library

Please change to something like "Isolating modules in the standard library". Most people don't have PEP numbers memorized; having to look up part of a document name sucks :) Same for "PEP 573"→"module state access"

recap user benefits

Limited API & PySide aren't really relevant for the stdlib. If you're going for a new document, keep the rationale/motivation limited to stdlib itself. The reasons should be e.g. "Py_Finalize() doesn't clear all Python objects at exit" (bpo-1635741).

Modules/xxmodule.c must be ported sooner or later

It's already ported, see Modules/xxlimited.c. xxmodule should stay as example of the original API, as long as that's supported. But as a tutorial that can be followed any time, it could work well.

[erlend-aasland]

Please change to something like "Isolating modules in the standard library". Most people don't have PEP numbers memorized; having to look up part of a document name sucks :) Same for "PEP 573"→"module state access"

Good points :) I've updated the post.

If you're going for a new document [...]

I think it will work better as a separate document. It'll be too comprehensive to add to PEP 630. Two relatively short documents are better than one large document, IMO :)

It's already ported, see Modules/xxlimited.c.

Ah, thanks!

[shihai1991]

Checked some global static vars:

• bool type use static type in bool_repr function: https://github.com/python/cpython/blob/62bf263a775f4444d8b5d5841cc09be3bd53e933/Objects/boolobject.c#L8-L9.
• Tracing reference. The refchain maintained in interpreter would be better if ignore the code churn? https://github.com/python/cpython/blob/62bf263a775f4444d8b5d5841cc09be3bd53e933/Objects/object.c#L93
• bpo-43009: Port curses capi pointer array to a struct.

[erlend-aasland]

I finally managed to allocate some time for this. I've gone back and forth a bit regarding the format and structure. So far, I think a short, informal, standalone document is fine for this. A draft is embedded in this post, but a PR is easier to review and collaborate with, so I figure I open a PR on this repo :)

Draft: Isolating Modules in the Standard Library

# Isolating modules in the standard library This document serves as a checklist for how to isolate a module in the standard library. As of Python 3.11, a lot of modules have been adapted to multi-phase initialisation, and a lot of standard library types have been converted from static to heap types. This process has not been straight-forward, and most of the issues relate to how heap types differ from static types. Some performance issues have also been encountered when converting global state to module state. Before undertaking this, you should familiarise yourself with the following PEPs: - [PEP 384](https://www.python.org/dev/peps/pep-0384/) - [PEP 489](https://www.python.org/dev/peps/pep-0489/) - [PEP 573](https://www.python.org/dev/peps/pep-0573/) - [PEP 630](https://www.python.org/dev/peps/pep-0630/) ## Part 1: Preparation 1. Open a discussion, either on the bug tracker or on Discourse. Involve the module maintainer and/or code owner. Explain the reason and rationale for the changes. 2. Identify global state performance bottlenecks, if there are such. Create a proof-of-concept implementation and measure the performance impact. `pyperf` is a good tool for benchmarking. 3. Create an implementation plan. For small modules with few types, a single PR may do the job. For larger modules with lots of types, and possibly also external library callbacks, multiple PR's will be needed. ## Part 2: Implementation Note: this is a suggested implementation plan, based on lessons learned with other modules. 1. Add Argument Clinic where possible; it enables you to easily use the defining class to fetch module state from type methods. 2. Prepare for module state; establish a module state `struct`, add an instance as a static global variable, and create helper stubs for fetching the module state. 3. Add relevant global variables to the module state `struct` and modify code that accesses the global state to use the module state helpers instead. This step may be broken into several PR's. 4. Convert heap types to static types. 5. Convert the global module state struct to true module state. 6. Implement multi-phase initialisation. Preferably, steps 4 through 6 should all land early in a single alpha development phase. ## Got'chas - All heap types **must fully implement the GC protocol**. See [bpo-42972](https://bugs.python.org/issue42972). - All standard library types should remain immutable. Heap types are mutable by default, static types are not. Use `Py_TPFLAGS_IMMUTABLE_TYPE` to retain immutability. See [bpo-43908](https://bugs.python.org/issue43908). - Static type with tp_new = NULL does not have public constructor, but heap type inherits constructor from base class. Make sure types that previously were impossible to instantiate, retain that feature; use `Py_TPFLAGS_DISALLOW_INSTANTIATION`. Add tests using `test.support.check_disallow_instantiation()`. See [bpo-43916](https://bugs.python.org/issue43916). - Use strong "back-refs" to the module object to ensure the module state pointer never outlives objects that access module state. Keep this in mind for external library callbacks that access module state.

[encukou]

Hi! Sorry about the delay, on my part this time. My life should be back to normal now :)

The draft looks OK. Perhaps it the goal should be to put it in the devguide, rather than a stand-alone document.

I think one thing missing from the guide is a more explicit note that not all types need to become heap types. If a type doesn't need the module state, it can be kept static. This might change depending on how/if GIL removal works, but it's too early in that project to know what will be needed.

[erlend-aasland]

The draft looks OK. Perhaps it the goal should be to put it in the devguide, rather than a stand-alone document.

The devguide sounds like a good place for such a document. That is, unless the SC thinks this calls for a stand-alone PEP, of course.

I think one thing missing from the guide is a more explicit note that not all types need to become heap types. [...] This might change depending on how/if GIL removal works, but it's too early in that project to know what will be needed.

So, we should note that as far as we know per now, not all types need become heap types, but that may change in the future.

If a type doesn't need the module state, it can be kept static.

Would that work correctly with multi-phase init?

[encukou]

So, we should note that as far as we know per now, not all types need become heap types, but that may change in the future.

It might change, but it also might not -- and not doing unnecessary changes helps limit the possible issues.

Would that work correctly with multi-phase init?

It should! If it doesn't, that's a pretty serious bug.

[erlend-aasland]

Ok, I'll update this in a couple of days hopefully. I'll update the PR on this repo. If the SC says an update to the devguide is enough, I'll create an issue and PR in that repo. For now, let's keep it here.

[encukou]

IMO, a PEP is in order. We need a PEP-like document, and we probably need SC approval, so let's go all the way. I'll take the existing document from the PR and work on it on it this week.

[erlend-aasland]

Great news!

[encukou]

PEP PR: https://github.com/python/peps/pull/2499