search

gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

ValueError on list comprehension with Python 3.8 #57

Closed mristin closed 4 years ago

mristin commented 4 years ago

Hi! The following program causes a ValueError when run with Python 3.8 and asttokens 1.1.15. The problem did not arise with Python 3.6.

import asttokens

text = '''@invariant(lambda self: all([" " not in n for n in self.parts]))
def dummy(): pass
'''
asttokens.ASTTokens(text, parse=True)

The error is:

Traceback (most recent call last):
  File "/home/marko/workspace/parquery/icontract/deleteme.py", line 6, in <module>
    asttokens.ASTTokens(text, parse=True)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/util.py", line 192, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'(' on line 1 col 28
alexmojaki commented 4 years ago

The latest version is 2.0.4

https://pypi.org/project/asttokens/#history https://github.com/gristlabs/asttokens/releases

mristin commented 4 years ago

Uh, my bad, please apologize. Let me please check if it works with the latest version before closing the issue.

mristin commented 4 years ago

@alexmojaki the issue is fixed in the newer version. Sorry again for my omission.