Add PyType_Freeze() function

[vstinner]

API: int PyType_Freeze(PyTypeObject *type)

• Make a type immutable: set the Py_TPFLAGS_IMMUTABLETYPE flag.
• All base classes of type must be immutable.
• On success, return 0. On error, set an exception and return -1.

The function is added to the stable ABI (version 3.14).

Feature requested by Cython to support the limited C API: https://github.com/python/cpython/issues/121654#issue-2406237843

Pull request: https://github.com/python/cpython/pull/122457

cc @encukou @da-woods

[vstinner]

Vote to add this API:

• [x] @encukou
• [x] @erlend-aasland
• [x] @mdboom
• [ ] @serhiy-storchaka
• [x] @vstinner
• [ ] @zooba

[encukou]

For me, this is a step toward breaking PyType_From* down into 3 steps:

• create a class with the minimum necessary info
• fill out the rest using setattr (or equivalent -- we'll need a function to apply an array of PyType_Slot to a class, but conceptually that would be convenience API for creating wrappers and calling setattr)
• make the class immutable, if necessary

This was sketched in markshannon/New-C-API-for-Python#4.

[zooba]

Can we also add a flag to make the original class unusable until it's frozen? It seems like we shouldn't let them escape in a partially-constructed state, and while that's usually something I'd happily leave to users to get right (and just document "don't let it escape or it may break one day"), in other cases we've ensured this with some kind of builder pattern.

I don't think we need an entirely new API for not-yet-frozen types. But perhaps a type slot that indicates "deferred construction" or something like that.

[vstinner]

Can we also add a flag to make the original class unusable until it's frozen?

What do you mean by "unusable"? As @encukou wrote, the pattern is to use regular APIs to prepare the class, like using "setattr". Do you want to use a new special API to prepare a class? It doesn't sound convenient.

[da-woods]

I don't think freezing classes is supposed to be a compulsory part of constructing the class. It just makes a mutable class immutable. But it'd be perfectly fine to let an unfrozen class escape if you intend it to be mutable.

[encukou]

Unlike tuple or str, here immutability is a bit on the instance. We don't want separate “initialization API” for what you can do to mutable types at runtime. If we do want to mark the original class unusable -- which operations would you disable?

I'm not too worried about letting a class escape while it's still mutable. Nearly all the danger with escaped objects is in someone “seeing” a partially-constructed instance, like a tuple with NULLs. Or someone might, say, use a still-to-be-modified string as a dict key, inadvertently caching the wrong hash. For types, AFAIK we're worried about “unauthorized” monkey-patching or instantiating. If someone does that to “escaped” objects, like ones they find in gc.get_objects, I think their warranty is void.

[zooba]

If we do want to mark the original class unusable -- which operations would you disable?

Instantiating it, mainly.

But really, I want to preserve our ability to later make significant changes to the type in PyType_Freeze that would be impossible if we can't assume that we've got the only reference to it. Things like producing a more efficient layout for the instantiated object (which is something people already have implemented, and it relies on assuming that a type is immutable).

We could achieve the same thing by documentation ("do not use the type if you intend to freeze it"). But if we can make it harder to do, then we're less likely to get people complaining when we change it (and if you think this is a hypothetical, go catch up on d.p.o for the latest complaints about changes and come back).

I don't think freezing classes is supposed to be a compulsory part of constructing the class.

I suggested adding a type slot that marks it as "deferred construction". Meaning that if that slot isn't set, PyType_Freeze would complain, and if it is set then instantiation would fail (or whatever other limit we decide should apply - PyType_Freeze would clear it). So it's not compulsory, but it is something that you would have to select at the initial construction time, not something that you could apply to any type at any time during runtime.

[encukou]

Instantiating it, mainly.

But at this level, “instatiating” is vague. One could call tp_new, or create an object with a compatible layout and set its ob_type (not something we want people to do, but quite possible with current API). And lots of things in between.

IMO, if a type supports something like optimized layouts, then that mechanism should check the invariant it needs -- for example, set a bit that says “the class was instantiated while immutable, disable optimization” (or raise a “instantiated a mutable class” error) from the allocator. Or from tp_new if that is when the class needs to be immutable.

[zooba]

IMO, if a type supports something like optimized layouts, then that mechanism should check the invariant it needs

I'm thinking of the nearly-hypothetical future where we do it for all types. At that stage, a limited API that prevents us from doing it broadly becomes a huge amount of dead weight, whereas we can make it easy to opt-in now and avoid that issue in the future.

Another alternative that I'd be happy with is if PyType_Freeze returns the frozen type, which may be the same pointer (incref'd) as the input, but may also be a different one. You get in just as much trouble if you're already using the pre-freeze type, but we're being explicit in our design that it's trouble, as opposed to merely hoping that people are doing everything right.

[encukou]

I'm thinking of the nearly-hypothetical future where we do it for all types. At that stage, a limited API that prevents us from doing it broadly becomes a huge amount of dead weight

We already have fully mutable types, so we can't apply such an optimization to all types. IMO, whenever we add the optimization, we can also add a “deferred construction” flag. (Existing libraries won't be able to use it, but I generally think that it's OK to require some churn if libraries want the latest optimizations.)

Also, PyType_From* isn't going anywhere, and I expect most classes will continue to use it for convenience. If we do manage to turn it into a convenience function for 3 calls to lower-level public API, it can start adding a “deferred construction” flag whenever it sees Py_TPFLAGS_IMMUTABLETYPE.

Another alternative that I'd be happy with is if PyType_Freeze returns the frozen type, which may be the same pointer (incref'd) as the input, but may also be a different one.

IMO, we're very unlikely to design a good API today that we can seamlessly switch to require copying types. Copying types is such a huge can of worms. (To start on the devilish details: do you expect the metaclass tp_new to be called?)

[zooba]

We already have fully mutable types, so we can't apply such an optimization to all types.

It turns out you can, because if you make a good enough guess as to what members a type will have, it's actually incredibly rare for more attributes to be added later. This was all analysed by Guido and others prior when implementing the optimisation (I believe Pinterest use it? Or possibly Instagram). The let down IIRC was the cost of de-optimising, but that's exactly the kind of thing that only takes one creative idea to fix.

On the other hand, the proposal here is to always create the most flexible types, and then tell some of them that they can't be changed. That absolutely relies on the assumption that we can't apply anything else in between (and only rely on a flag to block out some operations later).

All I'm proposing is the API to create the readonly type in a mutable state, and then it should be frozen before use. In the motivating case, they already know they want a readonly type, so it makes no difference there, and we ought to optimise the API for semantic intent (i.e. "this is a readonly type") and not optimisation intent (i.e. "internally this type is structured differently in a way that I neither see nor influence"). If we get the semantics right now, we don't have to worry about them in the future.

[encukou]

Given that Mark recommended this approach, I have doubts about it blocking optimizations; I'd rather let him chime in here.

But before that, a practical question: what, exactly, should a “deferred construction” flag prevent?

[zooba]

But before that, a practical question: what, exactly, should a “deferred construction” flag prevent?

Ideally, INCREF. Obviously that's not practical, but what it implies is that "nobody else should have a reference to this type before construction is complete (a.k.a. frozen)".

The more practical proposal would be to disallow instantiation, which is almost certainly unintended and also the most likely error to be made with a partially constructed type.

Mere references to the type (e.g. as a value in a dict) are unlikely to ever be problematic if the type goes from unfrozen to frozen. But once we've constructed an instance of that type, I expect it may be a problem to later change it from an unfrozen type to a frozen one. e.g. if another thread is already in one of its functions and we later decide that freezing types changes them to a linear layout instead of a dict.

The other side here is asking what does "freezing" a type mean? If it's just to avoid instance dicts, then I'm pretty sure that is trivial already (perhaps not via FromSpec, but easier done with a type slot flag than a new function). If it's to prevent all setattr, then I'd rather avoid the term "freezing" completely so we can save it for actual frozen types.

(namedtuple comes to mind as an example of an optimised frozen type, though that starts as a tuple and then adds names rather than the proposal here that starts with names and then aims to add restrictions. Adding __slots__ is another similar optimisation, but again, the slots are known at construction and applied before the type is usable in any form.)

[vstinner]

Let's say that a special type can have an optimized layout when it's frozen. I understand the desire to disallow instanciation before the type is frozen, it would be bad to have objects created before freeze with a layout A, and objects created created after freeze with layout B :-(

IMO such special type should be directly created as frozen, with Py_TPFLAGS_IMMUTABLETYPE, not using PyType_Freeze(). If the flag is set since the creation, objects with layout A cannot be created: optimized layout B is always used.

If tomorrow, we want to support such special type with PyType_Freeze() (defer the freeze step), I would suggest to wait until someone comes with a concrete implementation, so we can discuss about details.

Disallowing instanciation is complex since PyTypeObject and typeobject.c are complex. I would also prefer to not touch type_call() (not add a test) to avoid any performance degradation. Again, at least not touch it before someone comes with a more concrete implementation.

If tomorrow, someone comes with such "optimized layout" implementation, IMO Steve's design to "disallow instanciation until PyType_Freeze()" using a flag is a good idea. In the meanwhile, I don't think that it's needed.

[encukou]

The more practical proposal would be to disallow instantiation

But again, what is instantiation? tp_new can be easily bypassed.

Perhaps the choke point is tp_alloc? Would the concrete proposal be to add a flag that:

• causes PyType_GenericAlloc to raise, and
• is reset in PyType_Freeze?

(since type flags are running out, this could also be done by setting tp_alloc to a new function that always raises, which PyType_Freeze would special-case and replace?)

Then again, for this optimization, it seems we could also add a “has been instantianed yet” flag. tp_alloc (or wherever the “instantiation” check should be) could then:

• Check the bit; if it's clear and the class is already immutable then do the optimization. Otherwise, it'll remain un-optimized forever.
• Set the bit.

The other side here is asking what does "freezing" a type mean?

• Avoiding setattr on the class.
• Allowing immutable subclasses (which need their superclasses to be immutable)
• Conceptually, also setting the C-level fields and flags (with exceptions like refcount or subclasses). Of course we can't really prevent that, but we can have optimizations that depend on it.

If it's just to avoid instance dicts, then I'm pretty sure that is trivial already

Hm, instance dicts aren't really related to Py_TPFLAGS_IMMUTABLETYPE. This is about freezing the type object, not the instances.

[zooba]

But again, what is instantiation? tp_new can be easily bypassed.

The purpose is to explicitly discourage it enough that we can reserve the right to change it in the future. Historically, even though we've documented things as "don't use it like this", we've still respected code that misused it and avoided making changes that would break it.

I want enough of that kind of code to be broken up front, so that when it's time to change it, we aren't causing (many) new breaks.

this could also be done by setting tp_alloc to a new function that always raises, which PyType_Freeze would special-case and replace?

This works for me. I'm more concerned about the intent than the actual mechanism. I don't want to reach a point where we can't change this because Cython or whoever took a dependency on the fact that we didn't actively prevent it.