we-like-parsers / cpython

Here we work on integrating pegen into CPython; use branch 'pegen'
https://github.com/gvanrossum/pegen
Other
1 stars 0 forks source link

Segfault on invalid escape sequence in first line of file #201

Closed lysnikolaou closed 1 year ago

lysnikolaou commented 1 year ago

We've created an endless loop when there's an invalid escape sequence in the first line of a file.

Minimal reproducer:

In t.py:

f'\{'

When executing the file:

❯ ./python.exe t.py
/Users/lysnikolaou/repos/python/cpygen/tmp/t.py:1: DeprecationWarning: invalid escape sequence '\{'
/Users/lysnikolaou/repos/python/cpygen/tmp/t.py:1: DeprecationWarning: invalid escape sequence '\{'
/Users/lysnikolaou/repos/python/cpygen/tmp/t.py:1: DeprecationWarning: invalid escape sequence '\{'
...
zsh: segmentation fault  ./python.exe tmp/t.py

The problem is that when there's the sequence \{ in the literal part of an f-string we call warn_invalid_escape_sequence which tries to emit a warning and, in doing so, is using the tokenizer to scan the first line and get the file encoding. The exact stack trace is the following:

python.exe!_PyTokenizer_Get (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:2565)
python.exe!_PyTokenizer_FindEncodingFilename (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:2646)
python.exe!display_source_line_with_margin (/Users/lysnikolaou/repos/python/cpygen/Python/traceback.c:473)
python.exe!_Py_DisplaySourceLine (/Users/lysnikolaou/repos/python/cpygen/Python/traceback.c:581)
python.exe!show_warning (/Users/lysnikolaou/repos/python/cpygen/Python/_warnings.c:559)
python.exe!call_show_warning (/Users/lysnikolaou/repos/python/cpygen/Python/_warnings.c:583)
python.exe!warn_explicit (/Users/lysnikolaou/repos/python/cpygen/Python/_warnings.c:746)
python.exe!PyErr_WarnExplicitObject (/Users/lysnikolaou/repos/python/cpygen/Python/_warnings.c:1218)
python.exe!warn_invalid_escape_sequence (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:1342)
python.exe!tok_get_fstring_mode (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:2515)
python.exe!tok_get (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:2558)
python.exe!_PyTokenizer_Get (/Users/lysnikolaou/repos/python/cpygen/Parser/tokenizer.c:2565)
...