Column offsets in nested f-string expressions are wrong

[lysnikolaou]

Parsing the following:

expr = """
a = 10
f'{a * f"-{x()}-"}'"""

produces the following AST (not complete):

...
op=Mult(),
right=JoinedStr(
    values=[
        Constant(
            value="-",
            lineno=1,
            col_offset=5,
            end_lineno=1,
            end_col_offset=15,
        ),
        FormattedValue(
            value=Call(
                func=Name(
                    id="x",
                    ctx=Load(),
                    lineno=1,
                    col_offset=9,
                    end_lineno=1,
                    end_col_offset=10,
                ),
                args=[],
                keywords=[],
                lineno=1,
                col_offset=9,
                end_lineno=1,
                end_col_offset=12,
            ),
            conversion=-1,
            lineno=1,
            col_offset=5,
            end_lineno=1,
            end_col_offset=15,
        ),
        Constant(
            value="-",
            lineno=1,
            col_offset=5,
            end_lineno=1,
            end_col_offset=15,
        ),
                            ],
lineno=3,
col_offset=7,
end_lineno=3,
end_col_offset=17,
...

[lysnikolaou]

@pablogsal @gvanrossum Is there a way to fix this in a way that doesn't require too much effort? I can't think of something.

[lysnikolaou]

Line numbers were fixed by pre-feeding starting line and column numbers to the parser in https://github.com/we-like-parsers/cpython/pull/96/commits/991a634f420e2368fc64840b052430fde04a9ca3. There is still a bug with the column offsets, because the node shifting of the nested expression is executed before that of the outer one.

[pablogsal]

There is still a bug with the column offsets, because the node shifting of the nested expression is executed before that of the outer one.

Would a single node-shift of the whole f-string tree work? Or we do require two (ordered) shifts?

[lysnikolaou]

There is still a bug with the column offsets, because the node shifting of the nested expression is executed before that of the outer one.

Would a single node-shift of the whole f-string tree work? Or we do require two (ordered) shifts?

That might work, yeah! I'll work on it in a bit.