Three inconsistent module attributes

[53cd72ed-7768-4f34-9566-9995ab0e9dae]

BPO	37519
Nosy	@warsaw, @brettcannon, @ncoghlan, @ericvsmith, @ericsnowcurrently, @maggyero

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['interpreter-core', 'type-bug', 'library', '3.7'] title = 'Three inconsistent module attributes' updated_at = user = 'https://github.com/maggyero' ``` bugs.python.org fields: ```python activity = actor = 'eric.smith' assignee = 'none' closed = False closed_date = None closer = None components = ['Interpreter Core', 'Library (Lib)'] creation = creator = 'maggyero' dependencies = [] files = [] hgrepos = [] issue_num = 37519 keywords = [] message_count = 4.0 messages = ['347470', '347507', '347960', '347988'] nosy_count = 6.0 nosy_names = ['barry', 'brett.cannon', 'ncoghlan', 'eric.smith', 'eric.snow', 'maggyero'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue37519' versions = ['Python 3.7'] ```

[53cd72ed-7768-4f34-9566-9995ab0e9dae]

Analysis \========

In the next two sections showing the module attributes and corresponding spec attributes of imported modules and run modules, we notice the following rule (which is in accordance with this PEP-451 section <https://www.python.org/dev/peps/pep-0451/#attributes>_):

If a *spec* attribute is either ``None``, ``'built-in'`` or ``'frozen'``, then the corresponding *module* attribute is not set.

However we also notice three exceptions to this rule, that I think are unintended inconsistencies that should be corrected:

• module.__file__ is None (instead of being not set) for imported namespace packages;
• module.__cached__ is None (instead of being not set) for non-package modules run from the file system and run from standard input;
• module.__package__ is None (instead of being '') for non-package modules run from the file system and run from standard input.

The first exception was introduced recently (26 February 2018) by this pull request <https://github.com/python/cpython/pull/5481>_, which changed the module.__spec__.origin attribute from namespace to None (which I agree with as it avoids conflicting non-namespace-package modules named namespace with namespace packages) and the module.__file__ attribute from being unset to None (which I disagree with as it introduces an inconsistency and contradicts PEP-451).

Environment: CPython 3.7, MacOS 10.14.

Imported modules \================

Running the following code::

    import module

    print("MODULE")

for attr in ["__name__", "__file__", "__cached__", "__path__", "__package__", "__loader__"]:
    print(f"{attr}:", repr(getattr(module, attr, "not set")))

    print("SPEC")

    if hasattr(module, "__spec__"):
        if module.__spec__ is None:
            print("__spec__:", repr(module.__spec__))
        else:
            for attr in ["name", "origin", "cached", "submodule_search_locations", "parent", "loader"]:
                print(f"__spec__.{attr}:", repr(getattr(module.__spec__, attr)))
    else:
        print("__spec__: not set")

where module refers to:

• a non-package module (e.g., pathlib);
• a regular package (e.g., json);
• a namespace package;
• a built-in module (e.g., time);
• a frozen module (e.g., _frozen_importlib_external)

prints the following module attributes and corresponding spec attributes of the imported module:

• for a non-package module:

| MODULE | __name: 'pathlib' | __file: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py' | __cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pycache/pathlib.cpython-37.pyc' | __path: 'not set' | __package: '' | __loader: \<_frozen_importlibexternal.SourceFileLoader object at 0x1018896d8> | SPEC | \_spec.name: 'pathlib' | __spec.origin: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py' | __spec.cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pycache/pathlib.cpython-37.pyc' | __spec.submodule_searchlocations: None | \_spec.parent: '' | __spec.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x1018896d8>

• for a regular package:

| MODULE | __name: 'json' | __file: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py' | __cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/pycache/init.cpython-37.pyc' | __path: ['/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json'] | __package: 'json' | __loader: \<_frozen_importlibexternal.SourceFileLoader object at 0x10f9aa6d8> | SPEC | \_spec.name: 'json' | __spec.origin: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py' | __spec.cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/pycache/init.cpython-37.pyc' | __spec.submodule_searchlocations: ['/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json'] | \_spec.parent: 'json' | __spec.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x10f9aa6d8>

• for a namespace package:

| MODULE | __name: 'foobar' | __file: None | __cached: 'not set' | __path: _NamespacePath(['/Users/maggyero/foobar']) | __package: 'foobar' | __loader: \<_frozen_importlibexternal._NamespaceLoader object at 0x1074564a8> | SPEC | \_spec.name: 'foobar' | __spec.origin: None | __spec.cached: None | __spec.submodule_searchlocations: _NamespacePath(['/Users/maggyero/foobar']) | \_spec.parent: 'foobar' | __spec.loader: \<_frozen_importlib_external._NamespaceLoader object at 0x1074564a8>

• for a built-in module:

| MODULE | __name: 'time' | __file: 'not set' | __cached: 'not set' | __path: 'not set' | __package: '' | __loader: \<class '_frozenimportlib.BuiltinImporter'> | SPEC | \_spec.name: 'time' | __spec.origin: 'built-in' | __spec.cached: None | __spec.submodule_searchlocations: None | \_spec.parent: '' | __spec.loader: \<class '_frozen_importlib.BuiltinImporter'>

• for a frozen module:

| MODULE | __name: '_frozen_importlibexternal' | \_file: 'not set' | __cached: 'not set' | __path: 'not set' | __package: '' | __loader: \<class '_frozenimportlib.FrozenImporter'> | SPEC | \_spec.name: '_frozen_importlibexternal' | \_spec.origin: 'frozen' | __spec.cached: None | __spec.submodule_searchlocations: None | \_spec.parent: '' | __spec.loader: \<class '_frozen_importlib.FrozenImporter'>

Run modules \===========

Putting the following code::

    import sys

    print("MODULE")

for attr in ["__name__", "__file__", "__cached__", "__path__", "__package__", "__loader__"]:
    print(f"{attr}:", repr(getattr(sys.modules[__name__], attr, "not set")))

    print("SPEC")

    if hasattr(sys.modules[__name__], "__spec__"):
        if sys.modules[__name__].__spec__ is None:
            print("__spec__:", repr(sys.modules[__name__].__spec__))
        else:
            for attr in ["name", "origin", "cached", "submodule_search_locations", "parent", "loader"]:
                print(f"__spec__.{attr}:", repr(getattr(sys.modules[__name__].__spec__, attr)))
    else:
        print("__spec__: not set")

in:

• a module.py file for getting a module non-package module;
• a main.py file in a module directory with an init.py file for getting a module regular package;
• a main.py file in a module directory without an init.py file for getting a module namespace package

and running the code:

• from the file system (python3 module.py for a non-package module, python3 module/ for a package module);
• from standard input (cat module.py | python3 for a non-package module);
• from the module namespace (python3 -m module)

prints the following module attributes and corresponding spec attributes of the run module:

• for a non-package module:

$ python3 module.py	MODULE	__name: '__main'	__file__: 'module.py'	__cached__: None	__path__: 'not set'	__package__: None	__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x1051970b8>	SPEC	__spec__: None
$ cat module.py	python3
MODULE
__name: '__main'
__file__: '\<stdin>'
__cached__: None
__path__: 'not set'
__package__: None
__loader__: \<class '_frozen_importlib.BuiltinImporter'>
SPEC
__spec__: None
$ python3 -m module
MODULE
__name: '__main'
__file__: '/Users/maggyero/module.py'
__cached: '/Users/maggyero/pycache__/module.cpython-37.pyc'
__path__: 'not set'
__package__: ''
__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x1056b16d8>
SPEC
__spec__.name: 'module'
__spec__.origin: '/Users/maggyero/module.py'
__spec.cached: '/Users/maggyero/pycache__/module.cpython-37.pyc'
__spec__.submodule_search_locations: None
__spec__.parent: ''
__spec__.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x1056b16d8>

• for a regular package:

$ python3 module/	MODULE	__name: '__main'	__file: 'module/main__.py'	__cached: 'module/pycache/main__.cpython-37.pyc'	__path__: 'not set'	__package__: ''	__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x10826a550>	SPEC	__spec.name: '__main'	__spec.origin: 'module/main__.py'	__spec.cached: 'module/pycache/main__.cpython-37.pyc'	__spec__.submodule_search_locations: None	__spec__.parent: ''	__spec__.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x10826a550>
$ python3 -m module
MODULE
__name: '__main'
__file: '/Users/maggyero/module/main__.py'
__cached: '/Users/maggyero/module/pycache/main__.cpython-37.pyc'
__path__: 'not set'
__package__: 'module'
__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x10832d278>
SPEC
__spec.name: 'module.__main'
__spec.origin: '/Users/maggyero/module/main__.py'
__spec.cached: '/Users/maggyero/module/pycache/main__.cpython-37.pyc'
__spec__.submodule_search_locations: None
__spec__.parent: 'module'
__spec__.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x10832d278>

• for a namespace package:

$ python3 module/	MODULE	__name: '__main'	__file: 'module/main__.py'	__cached: 'module/pycache/main__.cpython-37.pyc'	__path__: 'not set'	__package__: ''	__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x107a06518>	SPEC	__spec.name: '__main'	__spec.origin: 'module/main__.py'	__spec.cached: 'module/pycache/main__.cpython-37.pyc'	__spec__.submodule_search_locations: None	__spec__.parent: ''	__spec__.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x107a06518>
$ python3 -m module
MODULE
__name: '__main'
__file: '/Users/maggyero/module/main__.py'
__cached: '/Users/maggyero/module/pycache/main__.cpython-37.pyc'
__path__: 'not set'
__package__: 'module'
__loader__: \<_frozen_importlib_external.SourceFileLoader object at 0x10fb69240>
SPEC
__spec.name: 'module.__main'
__spec.origin: '/Users/maggyero/module/main__.py'
__spec.cached: '/Users/maggyero/module/pycache/main__.cpython-37.pyc'
__spec__.submodule_search_locations: None
__spec__.parent: 'module'
__spec__.loader: \<_frozen_importlib_external.SourceFileLoader object at 0x10fb69240>

[brettcannon]

PEPs actually become historical documents once they are implemented, so could you please check what the official docs say in regards to this to see if there is an inconsistency in the semantics?

[53cd72ed-7768-4f34-9566-9995ab0e9dae]

@brett Cannon

PEPs actually become historical documents once they are implemented

Actually the inconsistency of the values of the 3 module attributes (__file__, __cached__ and __package__) is with respect to the other values within the current implementation (not only with respect to the values specified in PEP-451). Sorry if I did not explain this well. Let me detail:

For __file__, if you look at the current output of the above "Imported modules" section, you have:

• __file__: None, for an imported namespace package;
• __file__: 'not set', for an imported built-in module;
• __file__: 'not set', for an imported frozen module,

which is inconsistent: it should always be 'not set' when __file__ has no meaning.

For __cached__, if you look at the current output of the above "Run modules" section, you have:

• cached: None, for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3);
• path: 'not set', for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3),

which is inconsistent: it should always be 'not set' when __cached__ has no meaning, like it is already the case for __path__ and other module attributes.

For __package__, if you look at the current output of the above "Run modules" section, you have:

• package: None, for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3);
• package: '', for a non-package module run from the module namespace (python3 -m module) or a package run from the file system (python3 module/).

which is inconsistent: it should always be '' when there is no parent package for __package__ to refer to.

[ericvsmith]

While some of these might be inconsistent (I haven't really looked at it thoroughly yet), I think it might be problematic to change them at this point, since there's no doubt code out there that depends on the current behavior.

[picnixz]

Related: https://github.com/python/cpython/issues/64019