Open jolaf opened 5 years ago
I never found time to deal with @overload
, and pytypes might crash on loading stub files that contain it. So typeshed may be problematic. Further, stubfiles are searched in the same directory as the corresponding py-file or in the list of folders pytypes.stub_path
. I know, nowadays there is the specification PEP-561, but the pytypes.stub_path
-solution predates that specification. Once there was #41 but the proposed solution was withdrawn and I never found time to resume that issue. For now, pytypes.stub_path
must do it.
Problem with @overload
is a separate issue, I agree.
But the problem I hit was with file locations, not content.
For example, I have a typeshed
-like project of my own where I put the stubs I make for 3-rd party libraries. For exampe, I have stubs there for a simple one-module library netifaces
that is installed system-wide by root. So, the stubs for it turn on to be in ~/my-typeshed/netifaces/__init__.pyi
. However when I do import netifaces
, pytypes
for some reason looks for (I inserted prints to find out) netifaces.cpython-36m-x86_64-linux-gnu.pyi
.
pytypes.stub_path
wouldn't help with it, moreover, if stubs were in multiple files, and/or with submodules, I'd have to manually add each submodule to stub_path
.
So it seems there's a need for some reasonable logic for looking for stubs, something that could handle:
module/__init__.pyi
as well as module.pyi
;typeshed
hierarchy, like stdlib/3.6
, probably like a special option.Probably the logic mypy
uses would do, as it's already adopted by and familiar to those using mypy
which is probably the most important consumer of .pyi
stubs for now:
https://mypy.readthedocs.io/en/latest/running_mypy.html#how-mypy-handles-imports
cpython-36m-x86_64-linux-gnu
Interesting. Do you have an idea where this string comes from? Is a C-extension involved? Pytypes locates the stubfile based on the __file__
attribute of the module object. The full logic is a nightmare - I always hated the complexity required to get this seemingly simple matching right.
See these functions: https://github.com/Stewori/pytypes/blob/master/pytypes/util.py#L69 https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L107 https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L92 https://github.com/Stewori/pytypes/blob/master/pytypes/stubfile_manager.py#L152
Can you spot anything that causes cpython-36m-x86_64-linux-gnu
?
It probably comes from __file__
.
Yes, it's a C-extension module.
It's one more reason why it needs stubs. :)
$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import netifaces
>>> netifaces.__file__
'/usr/lib/python3/dist-packages/netifaces.cpython-36m-x86_64-linux-gnu.so'
Okay, perfect. Then we need better code to clean the filename, right? Can you suggest a genuine line of Python to accomplish this?
Probably https://github.com/Stewori/pytypes/blob/master/pytypes/util.py#L71 should be
bn = os.path.basename(module.__file__).partition('.')[0]
instead of rpartition
.
I wonder why I used rpartition
in first place. Can there be a case where the proper module name would contain a dot? Anyway, thanks for spotting this! If you like, file a PR to be accordingly credited.
Otherwise I can add this fix. However, I am currently working on another step towards 3.7 support. I prefer not to mix up work between distinct issues in my local branch, so I would fix this only after committing my current bunch of work. This might take until end of next week.
I wonder why use .__file__
in the first place, while there's .__name__
.
I remember there were issues with __name__
. It was certainly also my first choice. At very least, the main module has __main__
as its __name__
which is useless for finding the stubfile. There may have been further issues why __name__
was not reliable.
I wonder how mypy
resolves this. For example:
https://github.com/python/typeshed/tree/master/stdlib/3/os
Contains path.pyi
stubs for os.path
module.
However, on Windows:
C:\>python3
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.__name__
'ntpath'
>>> os.path.__file__
'C:\\Python36\\lib\\ntpath.py'
and on Ubuntu:
$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os.path
>>> os.path.__name__
'posixpath'
>>> os.path.__file__
'/usr/lib/python3.6/posixpath.py'
In other words, it seems that the name used to import a module can be different from the name visible through __name__
and __file__
.
Maybe in os
there is a platform-specific alias? However, querying such aliases would be tedious if possible at all. I guess the best solution would be a user-configurable lookup table for such "special cases". At least mypy does not have an issue with __main__
as itself takes that role.
I committed the change you suggest in https://github.com/Stewori/pytypes/issues/78#issuecomment-528857260. Does it fix that particular aspect of this issue for you? (Certainly the overall solution of this issue is WIP...)
Yep, thanks.
For example, I have
typeshed
, a project containing stubs for Python standard library. And I'd likepytypes
to check that my code calls standard library functions according to their annotations. How do I configurepytypes
to use the annotations intypeshed
?Whatever I tried didn't work.