search

we-like-parsers / pegen

PEG parser generator for Python
https://we-like-parsers.github.io/pegen/
MIT License
150 stars 32 forks source link

Pegen Crashes on f-strings in 3.12 #93

Closed Eric-Vin closed 10 months ago

Eric-Vin commented 11 months ago

Pegen seems to crash when trying to parse f-strings in Python 3.12. I'm guessing this is because Pegen is invoking the 3.12 tokenizer that now uses several new tokens for f-strings, not just STRING.

To reproduce I installed Pegen in 3.12 and ran python -m pegen data/python.gram, which resulted in the following error:

  File "<unknown>", line 1690
    self.raise_syntax_error_known_range(f"cannot assign to {a.string}", a, b)
                                        ^
SyntaxError: data/python.gram
MatthieuDartiailh commented 10 months ago

The Python parser needs to be updated to include the changes to the grammar in 3.12 indeed. Would you be interested in looking at the required changes ?

dfremont commented 10 months ago

Hello Matthieu, I've made a draft PR (#94) adding support for the 3.12 tokenizer and fixing the problem Eric mentioned above -- if you could take a look that would be great. Please note I haven't attempted to update the Python grammar for 3.12, since that's an orthogonal issue.

lysnikolaou commented 10 months ago

Closed in #94. Thanks @Eric-Vin for the report and @dfremont for the fix.