Python 3.12 changed how f-strings are handled and what they're capable of. It's likely because of this reason that the tokenizer module changed how it handles f-strings as well.
When finding escaped curly brackets in an f-string (e.g., f"{{") the tokenizer will only emit a single curly bracket and pretend the token ended one symbol earlier than the pair of brackets does. This PR basically hot-fixes those tokens to behave in a more consitent way by not swallowing the seconds bracket and by re-writing the token to end where the bracket pair actually ends.
Python 3.12 changed how f-strings are handled and what they're capable of. It's likely because of this reason that the
tokenizermodule changed how it handles f-strings as well.When finding escaped curly brackets in an f-string (e.g.,
f"{{") the tokenizer will only emit a single curly bracket and pretend the token ended one symbol earlier than the pair of brackets does. This PR basically hot-fixes those tokens to behave in a more consitent way by not swallowing the seconds bracket and by re-writing the token to end where the bracket pair actually ends.