Lexer exceeds maximum recursion depth if too many comments

[tsavola]

This happened when I commented out a long block of code (over 600 lines or so):

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "mypy/__main__.py", line 5, in <module>
    main(None)
  File "mypy/main.py", line 50, in main
    type_check_only(sources, bin_dir, options)
  File "mypy/main.py", line 94, in type_check_only
    python_path=options.python_path)
  File "mypy/build.py", line 210, in build
    result = manager.process(initial_states)
  File "mypy/build.py", line 427, in process
    next.process()
  File "mypy/build.py", line 759, in process
    tree = self.parse(self.program_text, self.path)
  File "mypy/build.py", line 856, in parse
    fast_parser=FAST_PARSER in self.manager.flags)
  File "mypy/parse.py", line 98, in parse
    tree = parser.parse(source)
  File "mypy/parse.py", line 147, in parse
    is_stub_file=self.is_stub_file)
  File "mypy/lex.py", line 170, in lex
    l.lex(string, first_line)
  File "mypy/lex.py", line 378, in lex
    map.get(c, default)()
  File "mypy/lex.py", line 751, in lex_break
    self.lex_indent()
  File "mypy/lex.py", line 698, in lex_indent
    self.lex_indent()
  File "mypy/lex.py", line 698, in lex_indent
    self.lex_indent()

  [...]

  File "mypy/lex.py", line 698, in lex_indent
    self.lex_indent()
  File "mypy/lex.py", line 697, in lex_indent
    self.lex_break()
  File "mypy/lex.py", line 735, in lex_break
    s = self.match(self.break_exp)
RecursionError: maximum recursion depth exceeded

[gvanrossum]

Eew! Good thing we're working on replacing the parser completely with a port of the ast module. (@ddfisher's project.)

[gvanrossum]

We don't have to do this assuming #1353 lands soon.

[gvanrossum]

I'm fixing this in the original code too, as a "plan B" in case typed_ast is delayed.