psf / black

The uncompromising Python code formatter
https://black.readthedocs.io/en/stable/
MIT License
37.6k stars 2.38k forks source link

Cannot parse multiline f-string containing multiline string #4337

Closed tarper24 closed 1 month ago

tarper24 commented 1 month ago

Describe the bug

Black fails to parse a file containing a multiline f-string which contains a multiline string within curly braces

To Reproduce

I created this MRC

f"""{'''
'''}"""

And just run black on the file:

$ black file.py

The resulting error is:

error: cannot format file.py: Cannot parse: 1:5: f"""{'''

Expected behavior

Successfully parse as it did in version <= 24.4.0

Environment

Additional context

Related to #4329

JelleZijlstra commented 1 month ago
% python3 -m blib2to3.pgen2.tokenize multil.py
1,0-1,4:    FSTRING_START   'f"""'
1,4-1,4:    FSTRING_MIDDLE  ''
1,4-1,5:    LBRACE  '{'
1,5-2,0:    FSTRING_MIDDLE  "'''\n"
2,0-2,3:    FSTRING_END "'''"
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 1115, in <module>
    tokenize(open(sys.argv[1]).readline)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 281, in tokenize
    tokenize_loop(readline, tokeneater)
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 288, in tokenize_loop
    for token_info in generate_tokens(readline):
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 680, in generate_tokens
    fstring_state.leave_fstring()
  File "/Users/jelle/py/black/src/blib2to3/pgen2/tokenize.py", line 541, in leave_fstring
    assert state == STATE_MIDDLE
AssertionError

It incorrectly emits FSTRING_MIDDLE even though the inner string is inside braces and isn't an f-string.

cc @tusharsadhwani