Allow built-in packages and submodules as well as top-level modules

[8aae074f-10b2-4f5a-94e9-0a8a01d6ede4]

BPO	1644818
Nosy	@loewis, @freddrake, @brettcannon, @terryjreedy, @ncoghlan, @vstinner, @encukou, @jd, @ericsnowcurrently
Files	import.diff: Allow importing built-in submodules (updated) test_submodule.py: Test case; to be placed in Lib/test xxsubmoduletest.c: Built-in submodule for regression test; to be placed in Modules test_builtin_submodule.py: More robust tests of expected package invariants

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['interpreter-core', 'type-feature'] title = 'Allow built-in packages and submodules as well as top-level modules' updated_at = user = 'https://bugs.python.org/mlobo' ``` bugs.python.org fields: ```python activity = actor = 'fdrake' assignee = 'none' closed = False closed_date = None closer = None components = ['Interpreter Core'] creation = creator = 'mlobo' dependencies = [] files = ['7739', '7740', '7741', '17126'] hgrepos = [] issue_num = 1644818 keywords = ['patch'] message_count = 14.0 messages = ['51802', '51803', '51804', '51805', '51806', '51807', '96588', '96590', '104436', '104506', '104507', '113445', '113471', '252374'] nosy_count = 12.0 nosy_names = ['loewis', 'fdrake', 'brett.cannon', 'collinwinter', 'terry.reedy', 'ncoghlan', 'vstinner', 'mlobo', 'Arfrever', 'petr.viktorin', 'jd', 'eric.snow'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue1644818' versions = ['Python 3.4'] ```

[8aae074f-10b2-4f5a-94e9-0a8a01d6ede4]

At the moment importing built-in submodules (in my case PyQt4.QtCore and PyQt4.QtGui) does not work. This seems to be because find_module in import.c checks only the module name (e.g. QtCore) against the built-in list, which should contain the full name (e.g. Python.QtCore) instead.

Also, the above check is performed after the code to check if the parent module is frozen, which would have already exited in that case.

By moving the is_builtin() check to earlier in find_module and using fullname instead of name, I can build PyQt4.QtCore and PyQt4.QtGui into the interpreter and import and use them with no problem whatsoever, even if their parent module (PyQt4) is frozen.

I have run the regression tests and everything seems Ok.

I am completely new to CPython development so it is quite possible that my solution is undesirable or that I have done something incorrectly. Please let me know if that is the case.

Finally, the attached patch is for Python-2.5, but I have checked it also applies to current svn trunk with only a one-line offset.

[cdc164a4-3067-4410-9a2e-ca8363400c3e]

This is the kind of thing you should bring up on the python-dev list (http://mail.python.org/mailman/listinfo/python-dev). Please subscribe to the list and send in a post with your problem and proposed solution. Thanks for your effort!

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Can you come up with a test case? Put the module in Modules/Setup, then have a test case that tries importing it. The test case should check whether the module is in builtin_module_names, and, if it is, try importing it.

(Ideally, it should only make the module builtin if debug is enabled, but I can't see how that can be achieved. Perhaps some trickery in config.c could do that, and #ifdefing out the body of the module if debug is disabled).

[8aae074f-10b2-4f5a-94e9-0a8a01d6ede4]

Ok, I have added a test case and it has actually allowed me to fix a small bug: I had tested the case where the parent of the submodule is a normal Python module and the case where it is a frozen module, but another small fix was needed to make it work if the parent is built-in itself.

Please be aware that I have only run the regression tests under Windows, as at the moment I don't have a suitable Unix box available.

Also, I have uploaded the test case file (test_submodule.py) separately, as I don't know how to convince SVN to put this file in the patch.

The new patch is against current trunk. File Added: import.diff

[8aae074f-10b2-4f5a-94e9-0a8a01d6ede4]

File Added: test_submodule.py

[8aae074f-10b2-4f5a-94e9-0a8a01d6ede4]

Argh, I've just realized I forgot to upload the built-in submodule itself. This obviously goes in Modules. File Added: xxsubmoduletest.c

[10895067-8d36-47bc-bc3b-f03803e78865]

Is there to chance to see this *bug* fixed someday?

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Is there to chance to see this *bug* fixed someday?

Please ask on python-dev. I may be willing to revive my five-for-one offer.

[ncoghlan]

I'll have a closer look at this tomorrow with the aim of getting it into 2.7b2.

(I'm inclined to agree with jd that this is just a bug in the existing implementation, hence the change in the issue type)

[ncoghlan]

OK, reassessing with brain fully engaged this time: the current patch is incorrect, and this request is more complicated than one might initially think :)

It appears that since the patch was originally tried out only on Windows, a Modules/Setup based system based system like mine can't even build a patched tree. The current incarnation of the Modules/makesetup script won't allow the use of a dotted name for a module named in Modules/Setup.

Anyway, I'm uploading a more fleshed out test case which explicitly details some of the module namespace invariants that built-in packages would need to support (and changing the issue type and title accordingly).

Even beyond these stricter tests, pkgutil and importlib would need to be checked to make sure they also support the new behaviour.

Since I can't build the patch as it currently stands, I don't know how well it actually fairs against the stronger set of invariants. However, just looking at the patch I'm pretty confident that it doesn't include the necessary work to make sure that the parent package actually looks like a package from the interpreter's point of view.

[ncoghlan]

Unassigning - this idea needs a lot more specification work to be done before actually implementing it becomes a good idea.

(e.g. IronPython and Jython should be looked at to see how they handle the naming schemes in the standard libraries for their respective runtime environments)

[terryjreedy]

I believe this is covered by the PEP-3003 3.2 change moratorium.

[brettcannon]

Import is explicitly exempt from the moratorium. But since the moratorium expires starting with Python 3.3 it really doesn't matter since this change will not go into 3.2.

[ncoghlan]

Petr, this could be an interesting issue to take a look at from the point of view of the builtin and extension module import changes in Python 3.5.

Given the extent of the import system changes across 3.3/4/5, it's even conceivable we may have made this work somewhere along the line. In that case we'd still need a new regression test to ensure it keeps working - we didn't even have a regression test to ensure extension module imports from inside a package worked, so we definitely don't have one for builtin submodules.