torch fails to load when an import hook is registered

[kevinushey]

First off -- thank you for importhook! It fills what feels like a surprising gap in Python's module import machinery.

Unfortunately, I'm seeing that some modules may fail to load when an importhook hook is registered. For example, here's a Bash script to reproduce when attempting to load torch:

#!/usr/bin/env bash

# create sample environment
python3 -m venv ~/.virtualenvs/torch-example
. ~/.virtualenvs/torch-example/bin/activate

# install importhook, torch
pip install importhook
pip install torch

# try importing torch with a hook active
cat <<EOF > script.py

import importhook
@importhook.on_import(importhook.ANY_MODULE)
def on_import(module):
    pass

import torch
EOF

python script.py

When I run this, I see:

Traceback (most recent call last):
  File "script.py", line 7, in <module>
    import torch
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/__init__.py", line 448, in <module>
    from .functional import *
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/functional.py", line 4, in <module>
    import torch.nn.functional as F
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/nn/__init__.py", line 1, in <module>
    from .modules import *
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/nn/modules/__init__.py", line 2, in <module>
    from .linear import Identity, Linear, Bilinear
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 6, in <module>
    from .. import functional as F
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/Users/kevinushey/.virtualenvs/torch-example/lib/python3.8/site-packages/torch/nn/functional.py", line 1099, in <module>
    threshold_ = _add_docstr(_VF.threshold_, r"""
AttributeError: module 'torch._VF' has no attribute 'threshold_'

It looks like the way that importhook patches the module finders + loaders interferes with torch in some odd way. Any idea what might be causing this?

[mazer-ai]

I just ran into the exact same problem -- before I start diving into this -- anybody else working on this?

[bollwyvl]

Another instance, here with IPython trying to import a magically-available method from pygments.lexers, presumably due to some of this cleverness:

  File "/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/lib/python3.10/_pyodide/_base.py", line 531, in eval_code_async
    await CodeRunner(
  File "/lib/python3.10/_pyodide/_base.py", line 359, in run_async
    await coroutine
  File "<exec>", line 10, in <module>
  File "/lib/python3.10/site-packages/pyolite/__init__.py", line 17, in <module>
    from .display import LiteStream
  File "/lib/python3.10/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/lib/python3.10/site-packages/pyolite/display.py", line 4, in <module>
    from IPython.core.displayhook import DisplayHook
  File "/lib/python3.10/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/lib/python3.10/site-packages/IPython/__init__.py", line 52, in <module>
    from .terminal.embed import embed
  File "/lib/python3.10/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/lib/python3.10/site-packages/IPython/terminal/embed.py", line 14, in <module>
    from IPython.core.magic import Magics, magics_class, line_magic
  File "/lib/python3.10/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/lib/python3.10/site-packages/IPython/core/magic.py", line 20, in <module>
    from . import oinspect
  File "/lib/python3.10/site-packages/importhook/loader.py", line 85, in exec_module
    mod = self.loader.exec_module(module, *args, **kwargs)
  File "/lib/python3.10/site-packages/IPython/core/oinspect.py", line 45, in <module>
    from pygments.lexers import PythonLexer
ImportError: cannot import name 'PythonLexer' from 'pygments.lexers' (/lib/python3.10/site-packages/pygments/lexers/__init__.py)

[bitterpanda63]

Yes, this issue is indeed because pygments.lexers relies on it being able to change sys.modules, This line of code generates the error. Currently working on a fix, might just be not changing sys.modules if we don't modify the module? Testing this out on a larger repo to see if this is indeed the fix.

[bitterpanda63]

For those interested I made a fix here: https://github.com/AikidoSec/firewall-python/pull/247, Since this project is not actively maintianed we keep our own version of importhook. But yeah I don't see a reason to create a PR right now