Support Python 3.12 f-strings in grammar actions

dfremont commented 10 months ago

As pointed out in #93, in Python 3.12 the PEP 701 changes to f-string tokenization cause Pegen to fail on grammars which contain f-strings in their actions (such as the Python grammar). This PR adds support for the new tokens to Pegen and changes the metagrammar to restore f-string support under 3.12 (while preserving support under older versions of Python).

Two things this PR does not currently do:

Fix broken tests under 3.12. I'm getting 28 failures currently, but as far as I can tell they don't have to do with the changes in this PR and are rather caused by other differences in the 3.12 tokenizer/parser. I can try to fix these later, but any help would be appreciated!
Update the example Python grammar (data/python.gram) for 3.12.

dfremont commented 10 months ago

Ah, that explains why it looked funny after I regenerated it!

dfremont commented 10 months ago

Thanks for looking this over, Pablo. I'll mark this ready for review in case you and Matthieu want to merge it without fixing the 3.12 test failures.

MatthieuDartiailh commented 10 months ago

The linting failures should be fixed before merging.

dfremont commented 10 months ago

Fixed!

MatthieuDartiailh commented 10 months ago

Thanks. The regeneration test still seems unhappy though.

dfremont commented 10 months ago

Whoops, I should have run tox. The problem is that the PythonParserGenerator gets its list of tokens from the token package, which prior to 3.12 doesn't include FSTRING_START etc. I've added them manually.

pablogsal commented 10 months ago

Thanks for the PR @dfremont !

we-like-parsers / pegen

Support Python 3.12 f-strings in grammar actions #94