stub file load works only across single file

[gaborbernat]

Problem presentation

The way the stub file implementation is done it will only work when all imports can be resolved using the Python files. A simple example when this is not the case is type aliases. For example imagine:

# util.pyi
from typing import Union

MagicParam = Union[int, float]

# magic.pyi
from .util import MagicParam

def a() -> MagicParam: ...

Loading magic.pyi will break with an import error of MagicParam as that is not defined inside the corresponding util.py. And given it's a type alias it should not be. As is used solely by the typing system.

The above example works with mypy, following the priority list described in PEP-561 resolution order.

Proposed solution

When loading stub files start with a clean import system, that first searches for pyi files, and if not present fallback to py. I have a prof of concept level of this here https://github.com/gaborbernat/prefer-stubs-python-import. If you think this request is reasonable I can make a PR for it. Thanks!

[Stewori]

Thanks for spotting this issue. You're more than welcome to help out, my current priority is on fixing Python 3.7 support.

Some things to keep in mind:

• For Python 2.7, we convert stubfiles to a 2.7 parsable form, i.e. using type comments and pass instead of .... These are potentially stored with .pyi2 suffix to avoid conflicts with Python 3 stubfiles in the same folder. See https://github.com/Stewori/pytypes#python-27-compliant-stubfiles.
• In some situations (something with running module initialization code in a module involving forward references in stubfiles) certain checks must be delayed until module import is finished. In that case, pytypes monkeypatches builtins.__import__ for detecting end of module import. (This shall be turned into an import hook, but that's still a todo.) Just pointing this out, hope it does not interfere with your approach. The monkeypatch is only applied if such a situation really comes up. Also see pytypes.import_hook_enabled for controlling this behavior.
• Currently pytypes also works with Jython. Keeping this compatibility valid can be tricky. I hope we can find a solution without breaking this support.
• pytypes.stubfile_manager applies a bit more magic than just loading the stubfile. When classes are used as types or type parameters it assures that not the class objects from the stubfile are used but instead the corresponding class objects from the actual module. It was tricky to get this right for nested classes, which lead to somewhat complicated code in stubfile_manager.py. It might take additional consideration to integrate your approach with this mechanism. See e.g. stubfile_manager._match_stub_type or stubfile_manager._match_classes.

I cannot tell right now if any of these notes applies to your approach (e.g. I never tried to use contextlib in Jython, maybe it just works). It's just things that might have to be considered. Good luck and feel free to ask for advice on pytypes internals etc if required!

[Stewori]

I hope my comments did not scare you away. Please feel free to go on with a PR, even if it wouldn't consider all the points above right out of the box. We can fix these things together step by step (as far as necessary at all).

[gaborbernat]

@Stewori did not scare me away but me realize my current implementation has conceptional problems. E.g. does not handle forward references at all...

[Stewori]

Note that pytypes has a mechanism to handle forward references: type_util.resolve_fw_decl which takes a module name as (optional) parameter, denoting a module where it looks for declarations of referenced variables. I'm not sure right now if it works with a stub file as module, but if not we should add it there.

[gaborbernat]

I gave this a good thought and decide to do this instead at mypy level :smiley: