importlib reload can fail with AttributeError if module removed from sys.path

[ned-deily]

BPO	21617
Nosy	@brettcannon, @ncoghlan, @ned-deily, @ericsnowcurrently
Files	test_sitecustomize.sh

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = [] title = 'importlib reload can fail with AttributeError if module removed from sys.path' updated_at = user = 'https://github.com/ned-deily' ``` bugs.python.org fields: ```python activity = actor = 'brett.cannon' assignee = 'none' closed = False closed_date = None closer = None components = [] creation = creator = 'ned.deily' dependencies = [] files = ['35416'] hgrepos = [] issue_num = 21617 keywords = [] message_count = 4.0 messages = ['219436', '321701', '321702', '321703'] nosy_count = 5.0 nosy_names = ['brett.cannon', 'ncoghlan', 'mforbes', 'ned.deily', 'eric.snow'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = None url = 'https://bugs.python.org/issue21617' versions = ['Python 3.4', 'Python 3.5'] ```

[ned-deily]

There are significant differences in behavior between Python 2.7, 3.3, and 3.4 (or current default) when using import reload() while manipulating sys.path. These differences cause unexpected behavior in the "Run Script" command of the TextMate 2 editor's Python bundle. TM2's Python "Run Script" is designed to work with either Python 2 or 2 interpreters. To do things like capture script stdout and stderr, the bundle launches a Python interpreter while prepending a directory with a custom sitecustomize module to sys.path. The TM2 sitecustomize then removes its directory from sys.path and uses reload to attempt to import any existing sitecustomize module. It then proceeds to do its TM2 customizations. In simplified pseudo-code:

# TM_DIRECTORY/sitecustomize.py
import sys
# remove this copy of sitecustomize from import path
if "TM_DIRECTORY" in sys.path:
    sys.path.remove("TM_DIRECTORY")
try:
    import sitecustomize    # get module reference
    if sys.version_info[0] >= 3:
        from imp import reload
    reload(sitecustomize)   # try to import another
                            # using altered path
except ImportError:
    pass                    # no other sitecustomize found
# do TM2 customizations here
# ...

this seems to work just fine with Python 2.7. With Python 3.3 the reload causes the originally imported sitecustomize to be used again, regardless of whether another sitecustomize exists on sys.path. For Python 3.4, 1d67eb1df5a9 for bpo-19851 changed reloading. Now for 3.4 (and default), if another sitecustomize exists on sys.path, the behavior is as expected as in 2.7: the other sitecustomize is imported by reload and executes. However, if there is no other sitecustomize, 3.4/default now gets an AttributeError exception in reload, rather than the expected ImportError. Enabling PYTHONVERBOSE shows:

File "./lib/python3.5/importlib/init.py", line 148, in reload _bootstrap._exec(spec, module) File "\<frozen importlib._bootstrap>", line 1083, in _exec AttributeError: 'NoneType' object has no attribute 'name'

And this is due to the call in reload() to _bootstrap._find_spec() returning None (Lib/importlib/init.py:147), presumably since there no longer is a sitecustomize. It looks like the following patch fixes the problem for this case:

diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py
--- a/Lib/importlib/__init__.py
+++ b/Lib/importlib/__init__.py
@@ -145,6 +145,9 @@
             pkgpath = None
         target = module
         spec = module.__spec__ = _bootstrap._find_spec(name, pkgpath, target)
+        if not spec:
+            msg = "module {} cannot be reloaded"
+            raise ImportError(msg.format(name), name=name)
         _bootstrap._exec(spec, module)
         # The module may have replaced itself in sys.modules!
         return sys.modules[name]

I'm not sure if this is a correct or sufficient change in the general case.

http://stackoverflow.com/questions/23962319/error-output-in-textmate-2-using-python-3-4

[e2f72cf1-d9db-45b6-89a2-e63c549db799]

What is the status of this issue? Is there a work-around? The 3.7 codebase raises an exception, but what is the correct solution for reloading a module that was loaded dynamically without modifying sys.path?

[e2f72cf1-d9db-45b6-89a2-e63c549db799]

It seems that a reasonable workaround (if you generate your own spec) is to call spec.loader.exec_module(mod). This seems to reload the module into the same namespace. (I was afraid that it might replace the namespace hence the desire to use importlib.reload() explicitly.)

[brettcannon]

If there's no spec, Michael, then your work-around is the next best option (and reloading never actually replaces the actual module object or it's dict/global namespace because we don't know who is holding on to a reference).