search

gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

IndexError on f((x)[:, 0]) #35

Closed ssbr closed 5 years ago

ssbr commented 5 years ago

I've had this for a while but only now found the time to boil down to a minimal reproducing case. :X

Reproducer:

import asttokens
asttokens.ASTTokens('f((x)[:, 0])', parse=True)

Running this file gives:

Traceback (most recent call last):
  File "/tmp/reproducer.py", line 2, in <module>
    asttokens.ASTTokens('f((x)[:, 0])', parse=True)
  File "asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "asttokens/util.py", line 192, in visit_tree
    ret = postvisit(current, par_value, value)
  File "asttokens/mark_tokens.py", line 87, in _visit_after_children
    first, last = self._expand_to_matching_pairs(first, last, node)
  File "asttokens/mark_tokens.py", line 142, in _expand_to_matching_pairs
    last = self._code.next_token(last_token)
  File "asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range