search

python / mypy

Optional static typing for Python
https://www.mypy-lang.org/
Other
18.27k stars 2.79k forks source link

hook for missing imports #8105

Open dvirtz opened 4 years ago

dvirtz commented 4 years ago

I have an Apache Airflow project and in that framework there's a massive use of dynamic module loading e.g. by manually creating modules from plugins and adding them to sys.modules.

I would like mypy to analyze my plugins but It cannot find those dynamic modules. One possible solution is to enable mypy plugins to provide a hook to resolve those imports the same way that is implemented in pylint: http://pylint.pycqa.org/projects/astroid/en/latest/extending.html#failed-import-hooks

Is there an easier way to achieve this?

trws commented 3 years ago

I was just searching for this same functionality or any other way around it. Did you ever come up with anything, or does anyone here have a workaround? So far we've tried the existing plugin interfaces, replacing the module with a class assigned into sys.modules, and various other things, but nothing seems to work.

There also doesn't seem to be a way to handle dynamically added module-level symbols for modules that can be loaded. Would a mechanism for this be welcomed in Mypy if it came to exist?

dvirtz commented 3 years ago

What I ended up doing is providing stub files for the airflow plugins e.g. stubs/airflow/operators/s3_operators.pyi

gvanrossum commented 3 years ago

Care to contribute those back to typeshed?

dvirtz commented 3 years ago

I'm not sure, it's very incomplete.

On Sat, 19 Dec 2020 at 21:09, Guido van Rossum notifications@github.com wrote:

Care to contribute those back to typeshed?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/python/mypy/issues/8105#issuecomment-748513076, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2ORDK5IM4W3YDLIR5LMDLSVT27TANCNFSM4JX3PQAA .

trws commented 3 years ago

Sadly I'm not sure I can express stubs for the two issues I'm trying to work around. The spack project uses injection into modules to create a DSL for expressing package information, and a fake import path to access package modules from other packages.

A simple example of the former would be the abi-dumper package. The symbols that are made available are partially determined by dependencies that aren't expressed in inheritance or otherwise, so to resolve the names in the package module would come down to something like a getattr hook on the module.

The second issue is anything under spack.pkg.* is loaded by a specialized loader and provided dynamically. We could convert these imports with a hook or similar, but I have yet to find a workaround for this other than ignoring it completely.

Admittedly both of these patterns are not terribly normal, nor are they kind to type-checking, but I was surprised to find that I could write a plugin to resolve the names were injected into a class but not with them injected into a module.