gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

Correctly mark slice token spans. #74

Closed ssbr closed 3 years ago

ssbr commented 3 years ago

Previous to this, the slice x[1:] would have the token span for 1, instead of 1:.

This is not the ideal way to do it -- I think ideally, mark_tokens should be using both col_offset and end_col_offset, in which case this would have worked automatically on Python 3.9. But this does get it working "as much as possible" in versions prior to 3.9.

In particular: while slices like 1: and :1 will now work correctly in Python 3.8-, the slice x[:] will still get attributed to x and not :. Oops. That issue does not exist in 3.9 (before or after this change).