gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

Adjust With nodes to include the "with " text #14

Closed FuegoFro closed 6 years ago

FuegoFro commented 6 years ago

Apparently in Python 2 the col_offset of With nodes starts after the "with " text. This behavior is confusing and inconsistent with Python 3, which has the offset start at the beginning of the "with".

For an example of the issue, run the following commands:

$ python3 -c 'import ast; print(ast.parse("with foo: pass").body[0].col_offset)'
0
$ python -c 'import ast; print(ast.parse("with foo: pass").body[0].col_offset)'
5

This diff adds a test that the With node includes the "with" text, and corresponding behavior to fix up the start location when running under Python 3.

dsagal commented 6 years ago

Thanks for this too. I felt I needed to experiment more, and found a case where it wasn't quite right. I added a test case for it and an alternative fix in #15 . Please take a look.

FuegoFro commented 6 years ago

This was superseded by #15