Crash on head when annotating

[jabdoa2]

Works fine on v1.0.2. On head this crashes:

pyannotate mpf/core/switch_controller.py 
Traceback (most recent call last):
  File "/usr/local/bin/pyannotate", line 11, in <module>
    load_entry_point('pyannotate', 'console_scripts', 'pyannotate')()
  File "/data/home/jan/cloud/flipper/src/pyannotate/pyannotate_tools/annotations/__main__.py", line 56, in main
    show_diffs=not args.quiet)
  File "/usr/lib/python3.5/lib2to3/main.py", line 63, in __init__
    super(StdoutRefactoringTool, self).__init__(fixers, options, explicit)
  File "/usr/lib/python3.5/lib2to3/refactor.py", line 698, in __init__
    super(MultiprocessRefactoringTool, self).__init__(*args, **kwargs)
  File "/usr/lib/python3.5/lib2to3/refactor.py", line 210, in __init__
    self.pre_order, self.post_order = self.get_fixers()
  File "/usr/lib/python3.5/lib2to3/refactor.py", line 255, in get_fixers
    fixer = fix_class(self.options, self.fixer_log)
  File "/usr/lib/python3.5/lib2to3/fixer_base.py", line 58, in __init__
    self.compile_pattern()
  File "/usr/lib/python3.5/lib2to3/fixer_base.py", line 67, in compile_pattern
    PC = PatternCompiler()
  File "/usr/lib/python3.5/lib2to3/patcomp.py", line 50, in __init__
    self.grammar = driver.load_grammar(grammar_file)
  File "/usr/lib/python3.5/lib2to3/pgen2/driver.py", line 120, in load_grammar
    logger.info("Generating grammar tables from %s", gt)
  File "/usr/lib/python3.5/logging/__init__.py", line 1279, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.5/logging/__init__.py", line 1414, in _log
    exc_info, func, extra, sinfo)
  File "/usr/lib/python3.5/logging/__init__.py", line 1384, in makeRecord
    sinfo)
  File "/usr/lib/python3.5/logging/__init__.py", line 269, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
  File "/usr/lib/python3.5/abc.py", line 191, in __instancecheck__
    return cls.__subclasscheck__(subclass)
  File "/usr/lib/python3.5/abc.py", line 226, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.5/abc.py", line 226, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.5/abc.py", line 226, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.5/typing.py", line 1081, in __subclasscheck__
    return issubclass(cls, self.__extra__)
  File "/usr/lib/python3.5/abc.py", line 226, in __subclasscheck__
    if issubclass(subclass, scls):
[...many more repetitions...]  
  File "/usr/lib/python3.5/typing.py", line 1081, in __subclasscheck__
    return issubclass(cls, self.__extra__)
  File "/usr/lib/python3.5/abc.py", line 226, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/usr/lib/python3.5/typing.py", line 1077, in __subclasscheck__
    if super().__subclasscheck__(cls):
  File "/usr/lib/python3.5/abc.py", line 197, in __subclasscheck__
    if subclass in cls._abc_cache:
RecursionError: maximum recursion depth exceeded

[gvanrossum]

Can't repro. ~Also your first sentence is~

[jabdoa2]

Sorry. Fixed that. Just bisected it. It starts crashing with this commit: https://github.com/dropbox/pyannotate/commit/01980b3ad607dd5a89c35a42226472b92ebbda65.

This is another PC with Ubuntu 16.04 and Python 3.5.2.

[gvanrossum]

Weird. Do you see anything in that commit that could have a bearing on logging initialization? I guess the most likely culprit is the added package typing_extensions. Could you try changing all the places where it says from typing_extensions import Text to from typing import Text? If that fixes it we know it's something in that module.

[gvanrossum]

Also does the contents of type_info.json matter? Or does it still crash when that's just []? And what's the output of pip list?

[gvanrossum]

Maybe @ilevkivskyi has any ideas?

[jabdoa2]

It works if I revert the occurrences of typing_extensions to typing.

The content of type_info.json does not matter. Also crashes if it is just [].

pip:

pip3 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
alabaster (0.7.10)
apigpio (0.0.1, /data/home/jan/cloud/flipper/src/apigpio)
appdirs (1.4.3)
apt-clone (0.2.1)
apt-xapian-index (0.47)
asciimatics (1.8.0)
astroid (1.5.3)
asyncio (3.4.3)
Babel (2.4.0)
beautifulsoup4 (4.4.1)
chardet (2.3.0)
command-not-found (0.3)
coverage (4.2)
Cython (0.23.1)
defer (1.0.6)
devscripts (2.16.2ubuntu3)
docutils (0.13.1)
dodgy (0.1.9)
future (0.16.0)
gitdb2 (2.0.2)
GitPython (2.1.5)
html5lib (0.999)
imagesize (0.7.1)
isort (4.2.15)
Jinja2 (2.9.6)
Kivy (1.10.0)
Kivy-Garden (0.1.4)
language-selector (0.1)
lazy-object-proxy (1.3.1)
lightdm-gtk-greeter-settings (1.2.1)
logilab-common (1.2.2)
lxml (3.5.0)
Magic-file-extensions (0.2)
MarkupSafe (1.0)
mccabe (0.6.1)
mock (2.0.0)
mpf (0.50.0.dev27, /data/home/jan/cloud/flipper/src/mpf)
mpf-mc (0.50.0.dev11, /data/home/jan/cloud/flipper/src/mpf-mc)
mpf-monitor (0.1.8, /data/home/jan/cloud/flipper/src/mpf-monitor)
mpmath (0.19)
mypy (0.540)
mypy-extensions (0.3.0)
olefile (0.44)
packaging (16.8)
PAM (0.4.2)
pbr (1.10.0)
pep8-naming (0.4.1)
Pillow (4.3.0)
pip (9.0.1)
prospector (0.12.7)
psutil (5.2.0)
pyannotate (1.0.2, /data/home/jan/cloud/flipper/src/pyannotate)
pycodestyle (2.0.0)
pycups (1.9.73)
pycurl (7.43.0)
pydocstyle (2.0.0)
pyfiglet (0.7.5)
pyflakes (1.6.0)
Pygments (2.2.0)
pygobject (3.20.0)
PyICU (1.9.2)
pylint (1.7.4)
pylint-celery (0.3)
pylint-common (0.2.5)
pylint-django (0.7.2)
pylint-flask (0.5)
pylint-plugin-utils (0.2.6)
pyparsing (2.2.0)
PyQt5 (5.7)
pyserial (3.3)
pyserial-asyncio (0.4, /data/home/jan/cloud/flipper/src/pyserial-asyncio)
python-afl (0.6)
python-apt (1.1.0b1)
python-debian (0.1.27)
python-systemd (231)
pytz (2017.2)
PyYAML (3.12)
requests (2.9.1)
requirements-detector (0.5.2)
ruamel.base (1.0.0)
ruamel.yaml (0.10.23)
setoptconf (0.2.0)
setuptools (35.0.2)
SimpleParse (2.2.0)
sip (4.18.1)
six (1.11.0)
smmap2 (2.0.3)
snowballstemmer (1.2.1)
Sphinx (1.6.3)
sphinx-rtd-theme (0.2.4)
sphinxcontrib-websupport (1.0.1)
ssh-import-id (5.5)
sympy (1.0)
typed-ast (1.1.0)
typing (3.6.2)
typing-extensions (3.6.2.1)
ubuntu-drivers-common (0.0.0)
ufw (0.35)
unattended-upgrades (0.1)
unittest-data-provider (1.0.1)
urllib3 (1.13.1)
usb-creator (0.3.0)
uvloop (0.8.1)
wcwidth (0.1.7)
wheel (0.29.0)
wrapt (1.10.11)
xkit (0.0.0)

[gvanrossum]

I wonder if a much simpler program that just tries to initialize logging (but imports typing_extensions first) would also crash.

[ilevkivskyi]

It is clear that the actual "problematic" line is isinstance(args[0], collections.Mapping). I think I understand why the crash happens, but I am not sure yet how to fix it in typing_extensions. The problem is that old version of typing calls issubclass(cls, self.__extra__) even if the latter is an ABC that will return to the same class via __subclasses__. This can be (and already is) fixed in recent typing.

A possible solution for typing_extensions is to define a custom metaclass inheriting from GenericMeta that will override __subclasscheck__. This could fix the issue, @jabdoa2 could you please open an issue on https://github.com/python/typing/issues?

[gvanrossum]

In the meantime I think I'll drop the typing_extensions dependency from pyannotate for now.