search

Stewori / pytypes

Typing-toolbox for Python 3 _and_ 2.7 w.r.t. PEP 484.
Apache License 2.0
200 stars 20 forks source link

Python >= 3.6.1 (maybe others too but not 3.6.0 or 3.5): Typecheck agent fails on re.compile() #65

Closed jolaf closed 5 years ago

jolaf commented 5 years ago

The following code:

import re
from pytypes import TypeChecker

with TypeChecker():
    re.compile('a')

fails as follows:

$ python3 Test.py 
/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py:2532: UserWarning: the system profiling hook has changed unexpectedly
  warn('the system profiling hook has changed unexpectedly')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/typecomment_parser.py", line 52, in _get_typestrings
    srclines = inspect.getsourcelines(obj)[0]
  File "/usr/lib/python3.6/inspect.py", line 955, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python3.6/inspect.py", line 768, in findsource
    file = getsourcefile(object)
  File "/usr/lib/python3.6/inspect.py", line 684, in getsourcefile
    filename = getfile(object)
  File "/usr/lib/python3.6/inspect.py", line 666, in getfile
    'function, traceback, frame, or code object'.format(object))
TypeError: <slot wrapper '__and__' of 'int' objects> is not a module, class, method, function, traceback, frame, or code object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Test.py", line 5, in <module>
    re.compile('a')
  File "/usr/lib/python3.6/re.py", line 233, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.6/re.py", line 302, in _compile
    if not (flags & DEBUG):
  File "/usr/lib/python3.6/enum.py", line 803, in __and__
    def __and__(self, other):
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py", line 2563, in __call__
    _check_caller_type(False, caller_level=self._caller_level_shift+1)
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py", line 2416, in _check_caller_type
    cllable, clss = _find_typed_base_method(cllable, clss)
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py", line 2131, in _find_typed_base_method
    if has_type_hints(util._actualfunc(fmeth)):
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py", line 657, in has_type_hints
    return _has_type_hints(func0)
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/type_util.py", line 686, in _has_type_hints
    tpStr = _get_typestrings(func, False)
  File "/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/typecomment_parser.py", line 54, in _get_typestrings
    srclines = inspect.getsourcelines(getattr(obj.__class__, obj.__name__))[0]
AttributeError: type object 'wrapper_descriptor' has no attribute '__and__'
Stewori commented 5 years ago

Is the re.compile failure closer to the root cause than datetime.strptime()? Or how is this change related?

jolaf commented 5 years ago

Yes, it's closer.

I've noticed that datetime.strptime() fails on re.compile() and reproduced the same problem on a more narrow test.

Stewori commented 5 years ago

It looks like I cannot reproduce this issue using the master branch. You are using the latest release, right? Master had some significant improvements, however the tedious progress in Python 3.7 support is blocking the next release. Maybe some of these improvements fixed it? Anyway, it would be good if you could confirm the one or other way.

jolaf commented 5 years ago

I'm using freshly built master branch on Ubuntu:

$ cat /etc/issue
Ubuntu 18.04.3 LTS \n \l

$ uname -a
Linux rudtzakhavas0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ git pull
Already up to date.

$ git show-branch 
[master] Fixes #63

$ sudo python3 setup.py install
running install
running bdist_egg
running egg_info
writing pytypes.egg-info/PKG-INFO
writing dependency_links to pytypes.egg-info/dependency_links.txt
writing entry points to pytypes.egg-info/entry_points.txt
writing requirements to pytypes.egg-info/requires.txt
writing top-level names to pytypes.egg-info/top_level.txt
writing manifest file 'pytypes.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/type_util.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/util.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/stubfile_2_converter.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/exceptions.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/typechecker.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/__init__.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/stubfile_manager.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/typelogger.py -> build/bdist.linux-x86_64/egg/pytypes
copying build/lib/pytypes/typecomment_parser.py -> build/bdist.linux-x86_64/egg/pytypes
byte-compiling build/bdist.linux-x86_64/egg/pytypes/type_util.py to type_util.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/util.py to util.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/stubfile_2_converter.py to stubfile_2_converter.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/exceptions.py to exceptions.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/typechecker.py to typechecker.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/stubfile_manager.py to stubfile_manager.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/typelogger.py to typelogger.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/pytypes/typecomment_parser.py to typecomment_parser.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pytypes.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
pytypes.__pycache__.stubfile_2_converter.cpython-36: module MAY be using inspect.findsource
pytypes.__pycache__.stubfile_manager.cpython-36: module references __file__
pytypes.__pycache__.typechecker.cpython-36: module MAY be using inspect.stack
pytypes.__pycache__.typelogger.cpython-36: module references __file__
pytypes.__pycache__.typelogger.cpython-36: module MAY be using inspect.findsource
pytypes.__pycache__.util.cpython-36: module references __file__
pytypes.__pycache__.util.cpython-36: module MAY be using inspect.stack
creating 'dist/pytypes-1.0b5.post20-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing pytypes-1.0b5.post20-py3.6.egg
removing '/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg' (and everything under it)
creating /usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg
Extracting pytypes-1.0b5.post20-py3.6.egg to /usr/local/lib/python3.6/dist-packages
pytypes 1.0b5.post20 is already the active version in easy-install.pth
Installing typestubs script to /usr/local/bin

Installed /usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg
Processing dependencies for pytypes==1.0b5.post20
Finished processing dependencies for pytypes==1.0b5.post20

$ python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytypes
>>> pytypes
<module 'pytypes' from '/usr/local/lib/python3.6/dist-packages/pytypes-1.0b5.post20-py3.6.egg/pytypes/__init__.py'>
Stewori commented 5 years ago

Okay, this seems to depend strictly on the python version. I can reproduce it with 3.6.1 but not with 3.5 and interestingly also not with 3.6.0.

Stewori commented 5 years ago

They must have changed something in the inspect module.

Stewori commented 5 years ago

In Python 3.6.1+, _get_typestrings is sometimes called with slot_wrapper objects which seems not to be the case up to 3.6.0. If it just returns None in slot_wrapper case that seems to fix the issue. But I'm not sure if that is the proper resolution. However, at least in your example, the slot_wrapper objects show up in addition to the function objects it receives in 3.6.0. So at least nothing is "lost" this way.

Stewori commented 5 years ago

Can a slot wrapper contain a type string anyway? I think slots are something implemented on C-level and can only carry type information via stub files. But I'm not sure if __slots__ can introduce slot wrappers backed by actual Python code.

Stewori commented 5 years ago

From https://stackoverflow.com/questions/10401935/python-method-wrapper-type/19545928#19545928 I conclude that slot wrappers need not be searched for type strings.

Stewori commented 5 years ago

This issue can be provoked by

import pytypes
pytypes.has_type_hints(int.__and__)

even in Python 2.7, however still not in Python 3 versions before 3.6.1. This is more convenient for testing than involving re into the tests.

jolaf commented 5 years ago

Confirmed fixed.