multiline f-strings with backslash continuation cause extra lines and misaligned coverage info in HTML report

[the-13th-letter]

Describe the bug

When a line of code ends on a multiline f-string that contains backslash continuation in the first line, the coverage HTML report inserts a synthetic line containing the backslash continuation's backslash at the same position (i.e. a line with a lot of whitespace, and then a backslash), following that line. This also increases the total line count. All following lines with coverage info are then misaligned, because the annotation gets assigned to the original line numbers, not the shifted ones.

To Reproduce

(Running Python 3.12.4 on Linux, in a fresh venv, using coverage 7.6.1 installed via pip install 'coverage === 7.6.1'.)

bug.py

prog_name = 'bug.py'
err_msg = f"""\
{prog_name}: ERROR: This is the first line of the error.
{prog_name}: ERROR: This is the second line of the error.
\
{prog_name}: ERROR: This is the third line of the error.
"""
if (lambda: None)():  # just some non-constant always-false predicate
    'This line intentionally is not covered.'
'This line is covered again.'

Shell commands

$ coverage run --branch python3.12 bug.py && coverage html
Wrote HTML report to htmlcov/index.html

Screenshot of the output:

Note the synthetic line 3, the misaligned partial/missing coloration on lines 8 and 9 (belonging to the code on lines 9 and 10), and the completely absent coloring of line 11. Note further that line 5 (6 in the screenshot) does not get its own synthetic line. Note finally that the syntax highlighting does not mark lines 2–8 as a string.

Expected behavior

I would expect the HTML rendering of the code bug.py to match the actual code of bug.py.

Additional context

• I could not reproduce this behavior if a b-string, u-string or r-string is used, but I could reproduce it with fr- and rf-strings.
• The other report types (JSON, XML, LCOV, text) are fine. In particular, the text-annotated version contains no synthetic lines, and all markers are at the right place.
• I tried switching to the Python trace function (run --timid) instead of the C trace function, and also tried the 3.12 sysmon core, but that didn't change anything. Which strengthens my belief that the collected data is fine, just the HTML reporter is not.

[nedbat]

@LiuYinCarl can you take a look at this? It sounds like the issue you fixed.

[LiuYinCarl]

@nedbat I just test it with version 7.6.1, but is works fine. Please give me a little more time to find the truth.

@the-13th-letter Can you use 7.6.0 to test it again with same file and same environment.

[LiuYinCarl]

@the-13th-letter I just test it with python3.12 and find some thing.

It seems like it has existed before version 7.6.0 and is only available on python3.12.

[LiuYinCarl]

python3.12 have some change in Lib/token.py, I find a method to solve the buy, I will make a PR later

[the-13th-letter]

@LiuYinCarl My apologies, I had already tested 7.6.0 and 7.0.0, but forgotten to mention that in the initial report. On the other hand, I had never thought to test it with different Python versions.

python3.12 have some change in Lib/token.py [...]

So it seems. The "what's new?" entry says it has to do with better f-string parsing (PEP 701).

[nedbat]

Fixed in commits https://github.com/nedbat/coveragepy/commit/515d99c6fd27101a53beb4e44ca35b0a0902c4d2 and https://github.com/nedbat/coveragepy/commit/a77bba7b80723624cc2b8636b8b9e2e63eaa4c5e. Thanks!