The type of cached objects is mutable

[serhiy-storchaka]

BPO	24912
Nosy	@malemburg, @gvanrossum, @brettcannon, @rhettinger, @ncoghlan, @pitrou, @vstinner, @larryhastings, @benjaminp, @ned-deily, @njsmith, @markshannon, @eltoder, @ericsnowcurrently, @serhiy-storchaka, @erlend-aasland
PRs	python/cpython#25714
Files	issue24912.patch issue24912-v2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/larryhastings' closed_at = created_at = labels = ['interpreter-core', 'type-bug', 'release-blocker'] title = 'The type of cached objects is mutable' updated_at = user = 'https://github.com/serhiy-storchaka' ``` bugs.python.org fields: ```python activity = actor = 'erlendaasland' assignee = 'larry' closed = True closed_date = closer = 'larry' components = ['Interpreter Core'] creation = creator = 'serhiy.storchaka' dependencies = [] files = ['40310', '40312'] hgrepos = [] issue_num = 24912 keywords = ['patch'] message_count = 69.0 messages = ['248981', '248982', '248983', '249019', '249180', '249189', '249198', '249345', '249350', '249351', '249354', '249357', '249360', '249363', '249369', '249388', '249392', '249393', '249396', '249397', '249398', '249438', '249446', '249450', '249455', '249456', '249457', '249467', '249468', '249470', '249471', '249473', '249483', '249484', '249489', '249495', '249499', '249500', '249502', '249504', '249505', '249506', '249509', '249510', '249511', '249604', '249607', '249618', '249623', '249685', '249854', '249869', '249871', '249888', '249889', '249890', '249891', '249893', '249930', '249931', '249934', '249938', '249940', '249943', '249944', '249948', '249954', '249979', '392411'] nosy_count = 18.0 nosy_names = ['lemburg', 'gvanrossum', 'brett.cannon', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'Arfrever', 'njs', 'Mark.Shannon', 'python-dev', 'eltoder', 'eric.snow', 'serhiy.storchaka', 'erlendaasland'] pr_nums = ['25714'] priority = 'release blocker' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue24912' versions = ['Python 3.5', 'Python 3.6'] ```

[gvanrossum]

We should have something for rc3, which is imminent. I vote for (1) and (2) if Serhiy thinks the patch is ready.

[serhiy-storchaka]

bpo-24912-v2.patch LGTM.

[gvanrossum]

Serhiy, can you commit it and prepare a PR for Larry?

[gvanrossum]

I'm creating the PR for Larry.

[gvanrossum]

PR: https://bitbucket.org/larry/cpython350/pull-requests/15/issue-24912-prevent-__class__-assignment/diff

[njsmith]

There's already a PR on bitbucket that I made, in case that's helpful. The only difference from the v2 patch are that I rephrased some of the comment in a more neutral way (based on feedback from Mark posted on the PR itself), and added a NEWS entry.

It does have a slightly more unsightly history graph due to the edit plus merging to resolve a conflict in NEWS.

[gvanrossum]

Oh. I feel dumb now. I guess I'll let Larry choose. If it's just a matter of comment text we can always improve it in 3.5.1.

[njsmith]

Sorry, you were too quick for me :-)

[markshannon]

Why has the change allowing the __class__ attribute of modules been merged?

It is not necessary (as I have stated repeatedly) and not known to be safe.

[larryhastings]

Because the BDFL asked that it be so.

[markshannon]

Well, there are important reasons why not to make the __class__ attribute of modules mutable.

First of all, there is no valid rationale.

How do know for sure that modifying the class of the sys or builtins module is not going to cause much the same problems the changing the class of ints did?

[markshannon]

Oh, and has anyone considered the potential impact this might have on Jython or PyPy? Modules are quite fundamental to the way that Python works.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 27cc5cce0292 by Guido van Rossum in branch '3.5': Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. https://hg.python.org/cpython/rev/27cc5cce0292

New changeset 1c55f169f4ee by Guido van Rossum in branch '3.5': Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5.0 -> 3.5) https://hg.python.org/cpython/rev/1c55f169f4ee

New changeset b045465e5dba by Guido van Rossum in branch 'default': Issue bpo-24912: Prevent __class__ assignment to immutable built-in objects. (Merge 3.5 -> 3.6) https://hg.python.org/cpython/rev/b045465e5dba

[gvanrossum]

Mark, please calm down. Modules are incredibly simple objects compared to classes -- and yet you can change the __class of a class. You can also directly modify the __dict of a module. You can shoot yourself in the foot phenomenally this way, but that's not the point. The point is not to violate *internal* constraints of the interpreter. Changing immutable objects was violating such a constraint. Changing a module's __class__ is not.

[markshannon]

Guido, I just think this change is misguided. The original rationale for the change just doesn't hold water.

If you want the change anyway, then fine, but this seems an odd way to introduce it.

[njsmith]

I didn't want to get further into the weeds of debating basic design points on the tracker while blocking rc3, but, for the record -- the proposal in msg249504 that one could keep the namespace of an old and new module object effectively in sync by defining __getattr on the new module doesn't work AFAICT. (I assume this is the basis of Mark's assertion that the change is unmotivated.) Going through __getattr creates an unacceptable performance hit on attribute access. You can avoid this by copying the namespace entries over (b/c __getattr is only called as a fallback when __dict lookup fails), but then you have the same problem as before of needing to keep things in sync. (And remember that module globals can change at any time -- the __globals__ pointer for any functions defined in that module will continue to point to the old namespace.)

Re: PyPy: last I heard their main complaint about __class__ assignment was that it was hard to restrict it to *only* handle the cases that CPython does. (They do have a concept of "heap types"; AFAICT it's just a flag that says "the CPython version of this type has some quirks that we need to check for and emulate".) Not sure about Jython, but I doubt they have a natural heap/non-heap type distinction either.

Thanks Guido for handling the patches!

[ericsnowcurrently]

While I recognize the practicality/purity argument here, I somewhat agree with Mark. Assigning to module.__class makes sense for the use case, but it does open up a number of negative possible side effects (e.g. changing sys.__class). Ideally it would be restricted to just the currently running module, but doing that through __class__ assignment is not a great idea. Having a dedicated builtin function or even syntax would make more sense.

In some ways I'm reminded of __metaclass in Python 2 class definitions. Something like that has come up before but didn't get a lot of momentum at the time. Ultimately I'd like to see a more explicit mechanism to meet the need that motivated the __class change here (as well as the replace-this-module-in-sys-modules technique). [1] While you can already accomplish this via a custom finder, that's not the most convenient tool for this sort of situation.

FWIW, if assigning to sys.modules[name].__class__ is the blessed way then we should make it official in the language reference. The same goes for modules replacing themselves in sys.modules. If we have reservations about making either official (which I for one do [2]) then we should work on a more appropriate official solution.

[1] It all boils down to customizing the current module's type from within the module rather than from outside (using a custom loader). [2] Neither spelling is very friendly.

[larryhastings]

Marking as closed for now. If we decide it's a problem we can reopen later.

[vstinner]

Update: a new Py_TPFLAGS_IMMUTABLETYPE flag was added in bpo-43908, and it's now checked rather than relying on Py_TPFLAGS_HEAPTYPE to decide if a type is "mutable" or not.