Expressions with format specifiers in f-strings have incorrect offsets

[alexcres]

def test(something):
    # "somethi" and "ng:b" are in different color.
    # it should be that "something" and ":b" are different color.
    return f'{something:b}'

[numirias]

Ha! Thanks for catching this. I'll look into it. (Unless someone else wants to supply a patch...)

[numirias]

I'm wondering if that's actually a bug in Python. Compare these two syntax trees:

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

The relevant part here is that the name a in the first f-string is positioned at

Name(lineno=1, col_offset=3, id='a', ctx=Load())

but in the second one at

Name(lineno=1, col_offset=1, id='a', ctx=Load())

although we're just adding a format specifier, not changing the position of the name in the source. To me it seems like the the column offset should remain at 3 chars. (Because f'{ are three characters.)

Can someone have a look at this as well? I'm tempted to file an issue in the Python tracker, but I may be overlooking something.

[numirias]

I filed a bug: https://bugs.python.org/issue35212

[alexcres]

Sorry I just went though python tutorial, can't help at all.

[sluongng]

hmm I am getting this as well :D... seems like the bug is still not fixed from Python side :(...

[numirias]

I'm currently experimenting with a parser based on tree-sitter instead of relying on Python's AST module. Once implemented, that would also eliminate bugs like this one.

[sluongng]

I'm currently experimenting with a parser based on tree-sitter instead of relying on Python's AST module. Once implemented, that would also eliminate bugs like this one.

I suggest putting this behind a feature flag and have a default value to be which ever one is more stable. Sure tree-sitter may work but I think Python AST may stick closer with the language to help with newer syntax(s) in the future

[p5a0u9l]

I'm seeing this as well - semshi is a great plugin, but this is really distracting. f-strings are all over my code. Looking into it a little..

Seems there's more going on beyond the case listed above. I think maybe it's not a bug in Python and your logic needs to detect format_spec is not None and have different behavior in that case.

Integrating a completely different solution (tree-sitter via built-in ast), imo, seems risky.

[sefechit]

I have the same problem, does everyone have it? :(

[jjmachan]

hey man, First of all, this is a great plugin!! Probably the top plugin. But, unfortunately, this bug is turning out to be a bummer. I use semshi daily since I code in nvim and this is messing up the beautiful highlightings :disappointed:.

I'll definitely try something out from my end but I'm a beginner so let me see.

Also hope your doing well with pandemic going on well wishes :smile:

[holmescharles]

More complaining about this bug. Any updates? This really makes me avoid f-strings, which is a shame.

Otherwise, love this plugin!!

[wookayin]

As of August 2021, this has been fixed in python upstream:

bpo-44885 (python/cpython#89048) -- fixed by python/cpython#27729.

• python 3.9.7+
• python 3.10.0 or higher