ikamensh / flynt

A tool to automatically convert old string literal formatting to f-strings
MIT License
689 stars 33 forks source link

Degree symbol causes flynt to think it sees EOF in multiline string #190

Open dkozinn opened 1 year ago

dkozinn commented 1 year ago

The following statement causes flynt to crash on what it thinks is an EOF in multiline string. Removing the degree symbol (°) results in output as expected:

flynt -v -s 'print("Feels like: {}°F".format(data["main"]["feels_like"]))'

Output from the above:

Skipping fstrings transform of file <code> due to ('EOF in multi-line statement', (2, 0)).
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/api.py", line 93, in fstringify_code
    new_code, changes = fstringify_code_by_line(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/code_editor.py", line 238, in fstringify_code_by_line
    return _transform_code(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/code_editor.py", line 278, in _transform_code
    ).edit()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/code_editor.py", line 71, in edit
    self.try_chunk(chunk)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/code_editor.py", line 129, in try_chunk
    if contains_comment(self.code_in_chunk(chunk)):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/flynt/utils/utils.py", line 132, in contains_comment
    for token in tokens:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/tokenize.py", line 523, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))
print("Feels like: {}°F".format(data["main"]["feels_like"]))

Output with degree symbol removed:

flynt -v -s 'print("Feels like: {}F".format(data["main"]["feels_like"]))'
print(f"Feels like: {data['main']['feels_like']}F")

As you'd expect, the results are the same when processing that statement as part of a file.