search

we-like-parsers / cpython

Here we work on integrating pegen into CPython; use branch 'pegen'
https://github.com/gvanrossum/pegen
Other
1 stars 0 forks source link

Make FSTRING_END contain the closing quote (?) #206

Closed lysnikolaou closed 1 year ago

lysnikolaou commented 1 year ago

While this is an implementation detail, Guido correctly pointed out that is really asymmetrical that the closing quote is not part of the FSTRING_END token. My feeling is that we should change the implementation and amend the PEP accordingly. What do you think?

CC @pablogsal @isidentical

pablogsal commented 1 year ago

My feeling is that we should change the implementation and amend the PEP accordingly. What do you think?

Note we don't need to change the implementation of the C tokenizer, although we should check if the change makes things easier for sure. The key is that the public interface (the Python tokenizer) should for sure be symmetric.

I will change the PEP in the following days

@isidentical thoughts?

isidentical commented 1 year ago

Note we don't need to change the implementation of the C tokenizer, although we should check if the change makes things easier for sure. The key is that the public interface (the Python tokenizer) should for sure be symmetric.

This will also simplify the untokenization since if we were to stick with the current format, the untokenizer would have needed to keep track of the bound start token to match the closing quotes. Now we can simply use the FSTRING_END's value without any sort of additional state in the untokenizer.

For the C one, I think it would look nice if we either change it to match or not mention about its behavior at all at the PEP.

pablogsal commented 1 year ago

Heads up! I have written the new specification here:

https://github.com/python/peps/pull/2974