search

gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

failures with cpython master #55

Closed tacaswell closed 4 years ago

tacaswell commented 4 years ago

There have been changes to the AST upstream in cpython are breaking asttokens

====================================================================== short test summary info ======================================================================
FAILED tests/test_astroid.py::TestAstroid::test_complex_slice_and_parens - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - AssertionError: 'Slice(\n   lower=None,\n   upper=None,\n   step=None)' != "Name(name='d')"
FAILED tests/test_astroid.py::TestAstroid::test_nonascii - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_slices - AssertionError: 'Slice(\n   lower=None,\n   upper=Const(value=3),\n   step=None)' != 'Const(value=3)'
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - AssertionError: "Slice(\n   lower=None,\n   upper=BinOp(\n[245 chars]one)" != "BinOp(\n   op='-',\n ...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_assignment_expressions - ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_complex_slice_and_parens -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_decorators - ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture8 - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture9 - ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_keyword_arg_only - ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_mark_tokens_simple - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_nonascii -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_slices -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules - ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13

In addition there seems to have been changes that break asttokens in a way that breaks stack-data which breaks IPython that don't look like they are capture in the tests:

bleeding) ✘  ~/source/other_source/asttokens [master {origin/master}|✔ ]
jupiter@17:52  ➤  ipython
Python 3.9.0a5+ (heads/master:799d7d61a9, Apr  6 2020, 17:38:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: 1/0                                                                                                                                                          
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-9e1622b385b6>", line 1, in <module>
    1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:
<snipped>
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__
    self.asttokens()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens
    return ASTTokens(
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9

Full pytest output:

``` (bleeding) ✘ ~/source/other_source/asttokens [master {origin/master}|✔ ] jupiter@17:43 ➤ pytest ======================================================================== test session starts ======================================================================== platform linux -- Python 3.9.0a5+, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /home/tcaswell/source/other_source/asttokens, inifile: setup.cfg plugins: cov-2.8.1, rerunfailures-9.0, timeout-1.3.4, forked-1.1.3, xdist-1.31.0, benchmark-3.2.3 collected 107 items tests/test_astroid.py .........F...s...............F....F.....F..F... [ 43%] tests/test_asttokens.py ...... [ 49%] tests/test_line_numbers.py ... [ 52%] tests/test_mark_tokens.py ...F.....F..F...............FF.F.FF.....F..F... [ 96%] tests/test_util.py .... [100%] ============================================================================= FAILURES ============================================================================== _____________________________________________________________ TestAstroid.test_complex_slice_and_parens _____________________________________________________________ self = def test_complex_slice_and_parens(self): source = 'f((x)[:, 0])' > self.create_mark_checker(source) tests/test_mark_tokens.py:597: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:90: in verify_all_nodes rebuilt_node = test_case.parse_snippet(text, node) tests/test_mark_tokens.py:735: in parse_snippet return self.module.parse('_\n(' + text + ')').body[1].value ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:275: in parse return builder.string_build(code, modname=module_name, path=path) ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:140: in string_build module = self._data_build(data, modname, path) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , data = '_\n(:, 0)', modname = '', path = None def _data_build(self, data, modname, path): """Build tree node from data and add some informations""" try: node = _parse(data + "\n") except (TypeError, ValueError, SyntaxError) as exc: > raise exceptions.AstroidSyntaxError( "Parsing Python code failed:\n{error}", source=data, modname=modname, path=path, error=exc, ) from exc E astroid.exceptions.AstroidSyntaxError: Parsing Python code failed: E invalid syntax (, line 2) ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:168: AstroidSyntaxError _____________________________________________________________________ TestAstroid.test_fixture9 _____________________________________________________________________ self = > def test_fixture9(self): self.verify_fixture_file('astroid/module2.py') tests/test_mark_tokens.py:172: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:134: in verify_fixture_file tested_nodes = m.verify_all_nodes(self) tests/tools.py:97: in verify_all_nodes test_case.assertEqual( E AssertionError: 'Slice(\n lower=None,\n upper=None,\n step=None)' != "Name(name='d')" E + Name(name='d')- Slice( E - lower=None, E - upper=None, E - step=None) _____________________________________________________________________ TestAstroid.test_nonascii _____________________________________________________________________ self = def test_nonascii(self): # Test of https://bitbucket.org/plas/thonny/issues/162/weird-range-marker-crash-with-non-ascii # Only on PY3 because Py2 doesn't support unicode identifiers. for source in ( "sünnikuupäev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]", "sünnikuupaev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]"): > m = self.create_mark_checker(source) tests/test_mark_tokens.py:286: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:90: in verify_all_nodes rebuilt_node = test_case.parse_snippet(text, node) tests/test_mark_tokens.py:735: in parse_snippet return self.module.parse('_\n(' + text + ')').body[1].value ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:275: in parse return builder.string_build(code, modname=module_name, path=path) ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:140: in string_build module = self._data_build(data, modname, path) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , data = '_\n(0:1)', modname = '', path = None def _data_build(self, data, modname, path): """Build tree node from data and add some informations""" try: node = _parse(data + "\n") except (TypeError, ValueError, SyntaxError) as exc: > raise exceptions.AstroidSyntaxError( "Parsing Python code failed:\n{error}", source=data, modname=modname, path=path, error=exc, ) from exc E astroid.exceptions.AstroidSyntaxError: Parsing Python code failed: E invalid syntax (, line 2) ../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:168: AstroidSyntaxError ______________________________________________________________________ TestAstroid.test_slices ______________________________________________________________________ self = def test_slices(self): # Make sure we don't fail on parsing slices of the form `foo[4:]`. source = "(foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::2, :], [a[:]][::-1])" > m = self.create_mark_checker(source) tests/test_mark_tokens.py:230: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:97: in verify_all_nodes test_case.assertEqual( E AssertionError: 'Slice(\n lower=None,\n upper=Const(value=3),\n step=None)' != 'Const(value=3)' E + Const(value=3)- Slice( E - lower=None, E - upper=Const(value=3), E - step=None) ___________________________________________________________________ TestAstroid.test_sys_modules ____________________________________________________________________ self = def test_sys_modules(self): """ Verify all nodes on source files obtained from sys.modules. This can take a long time as there are many modules, so it only tests all modules if the environment variable ASTTOKENS_SLOW_TESTS has been set. """ modules = list(sys.modules.values()) if not os.environ.get('ASTTOKENS_SLOW_TESTS'): modules = modules[:20] start = time() for module in modules: # Don't let this test (which runs twice) take longer than 13 minutes # to avoid the travis build time limit of 30 minutes if time() - start > 13 * 60: break try: filename = inspect.getsourcefile(module) except TypeError: continue if not filename: continue filename = os.path.abspath(filename) print(filename) try: with io.open(filename) as f: source = f.read() except OSError: continue # Astroid fails with a syntax error if a type comment is on its own line if self.is_astroid_test and re.search(r'^\s*# type: ', source, re.MULTILINE): print('Skipping', filename) continue > self.create_mark_checker(source) tests/test_mark_tokens.py:639: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:97: in verify_all_nodes test_case.assertEqual( E AssertionError: "Slice(\n lower=None,\n upper=BinOp(\n[245 chars]one)" != "BinOp(\n op='-',\n left=Call(\n [172 chars]f'))" E + BinOp( E - Slice( E - lower=None, E - upper=BinOp( E - op='-', E ? --- E + op='-', E - left=Call( E ? --- E + left=Call( E - func=Name(name='len'), E ? --- E + func=Name(name='len'), E - args=[Attribute( E ? --- E + args=[Attribute( E - attrname='__name__', E ? --- E + attrname='__name__', E - expr=Name(name='module'))], E ? --- E + expr=Name(name='module'))], E - keywords=None), E ? --- E + keywords=None), E - right=Name(name='cut_off')), E ? --- -- E + right=Name(name='cut_off'))- step=None) ----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------ /home/tcaswell/.pybuild/bleeding/lib/python3.9/importlib/_bootstrap.py ____________________________________________________________ TestMarkTokens.test_assignment_expressions _____________________________________________________________ self = def test_assignment_expressions(self): # From https://www.python.org/dev/peps/pep-0572/ > self.create_mark_checker(""" # Handle a matched regex if (match := pattern.search(data)) is not None: # Do something with match pass # A loop that can't be trivially rewritten using 2-arg iter() while chunk := file.read(8192): process(chunk) # Reuse a value that's expensive to compute [y := f(x), y**2, y**3] # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None] y0 = (y1 := f(x)) # Valid, though discouraged foo(x=(y := f(x))) # Valid, though probably confusing def foo(answer=(p := 42)): # Valid, though not great style ... def foo(answer: (p := 42) = 5): # Valid, but probably never useful ... lambda: (x := 1) # Valid, but unlikely to be useful (x := lambda: 1) # Valid lambda line: (m := re.match(pattern, line)) and m.group(1) # Valid if any((comment := line).startswith('#') for line in lines): print("First comment:", comment) if all((nonblank := line).strip() == '' for line in lines): print("All lines are blank") partial_sums = [total := total + v for v in values] """) tests/test_mark_tokens.py:678: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=1, string='y0', start=(17, 0), end=(17, 2), line='y0 = (y1 := f(x)) # Valid, though discouraged\n', index=92, startpos=435, endpos=437) tok_type = 1, tok_str = 'x' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1 asttokens/util.py:56: ValueError ___________________________________________________________ TestMarkTokens.test_complex_slice_and_parens ____________________________________________________________ self = def test_complex_slice_and_parens(self): source = 'f((x)[:, 0])' > self.create_mark_checker(source) tests/test_mark_tokens.py:597: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:90: in verify_all_nodes rebuilt_node = test_case.parse_snippet(text, node) tests/test_mark_tokens.py:735: in parse_snippet return self.module.parse('_\n(' + text + ')').body[1].value _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ source = '_\n(:, 0)', filename = '', mode = 'exec' def parse(source, filename='', mode='exec', *, type_comments=False, feature_version=None): """ Parse the source into an AST node. Equivalent to compile(source, filename, mode, PyCF_ONLY_AST). Pass type_comments=True to get back type comments where the syntax allows. """ flags = PyCF_ONLY_AST if type_comments: flags |= PyCF_TYPE_COMMENTS if isinstance(feature_version, tuple): major, minor = feature_version # Should be a 2-tuple. assert major == 3 feature_version = minor elif feature_version is None: feature_version = -1 # Else it should be an int giving the minor version for 3.x. > return compile(source, filename, mode, flags, _feature_version=feature_version) E File "", line 2 E (:, 0) E ^ E SyntaxError: invalid syntax ../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError __________________________________________________________________ TestMarkTokens.test_decorators ___________________________________________________________________ self = def test_decorators(self): # See https://bitbucket.org/plas/thonny/issues/49/range-marker-fails-with-decorators source = textwrap.dedent(""" @deco1 def f(): pass @deco2(a=1) def g(x): pass @deco3() def g(x): pass """) > m = self.create_mark_checker(source) tests/test_mark_tokens.py:491: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=6, string='', start=(12, 0), end=(12, 0), line='', index=49, startpos=80, endpos=80), tok_type = 1, tok_str = 'a' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1 asttokens/util.py:56: ValueError ___________________________________________________________________ TestMarkTokens.test_fixture8 ____________________________________________________________________ self = > def test_fixture8(self): self.verify_fixture_file('astroid/module.py') tests/test_mark_tokens.py:171: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:133: in verify_fixture_file m = self.create_mark_checker(source, verify=False) tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=1, string='autre', start=(53, 12), end=(53, 17), line=' autre = [a for (a, b) in MY_DICT if b]\n', index=224, startpos=967, endpos=972) tok_type = 1, tok_str = 'val' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13 asttokens/util.py:56: ValueError ___________________________________________________________________ TestMarkTokens.test_fixture9 ____________________________________________________________________ self = > def test_fixture9(self): self.verify_fixture_file('astroid/module2.py') tests/test_mark_tokens.py:172: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:133: in verify_fixture_file m = self.create_mark_checker(source, verify=False) tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=1, string='e', start=(98, 0), end=(98, 1), line='e = d[a:b:c]\n', index=476, startpos=1233, endpos=1234), tok_type = 1, tok_str = 'file' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1 asttokens/util.py:56: ValueError _______________________________________________________________ TestMarkTokens.test_keyword_arg_only ________________________________________________________________ self = def test_keyword_arg_only(self): # See https://bitbucket.org/plas/thonny/issues/52/range-marker-fails-with-ridastrip-split source = "f(x=1)\ng(a=(x),b=[y])" > m = self.create_mark_checker(source) tests/test_mark_tokens.py:460: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=4, string='', start=(2, 14), end=(2, 15), line='', index=21, startpos=21, endpos=21), tok_type = 1, tok_str = 'x' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15 asttokens/util.py:56: ValueError ______________________________________________________________ TestMarkTokens.test_mark_tokens_simple _______________________________________________________________ self = def test_mark_tokens_simple(self): source = tools.read_fixture('astroid', 'module.py') > m = self.create_mark_checker(source) tests/test_mark_tokens.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=1, string='autre', start=(53, 12), end=(53, 17), line=' autre = [a for (a, b) in MY_DICT if b]\n', index=224, startpos=967, endpos=972) tok_type = 1, tok_str = 'val' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13 asttokens/util.py:56: ValueError ___________________________________________________________________ TestMarkTokens.test_nonascii ____________________________________________________________________ self = def test_nonascii(self): # Test of https://bitbucket.org/plas/thonny/issues/162/weird-range-marker-crash-with-non-ascii # Only on PY3 because Py2 doesn't support unicode identifiers. for source in ( "sünnikuupäev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]", "sünnikuupaev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]"): > m = self.create_mark_checker(source) tests/test_mark_tokens.py:286: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:90: in verify_all_nodes rebuilt_node = test_case.parse_snippet(text, node) tests/test_mark_tokens.py:735: in parse_snippet return self.module.parse('_\n(' + text + ')').body[1].value _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ source = '_\n(0:1)', filename = '', mode = 'exec' def parse(source, filename='', mode='exec', *, type_comments=False, feature_version=None): """ Parse the source into an AST node. Equivalent to compile(source, filename, mode, PyCF_ONLY_AST). Pass type_comments=True to get back type comments where the syntax allows. """ flags = PyCF_ONLY_AST if type_comments: flags |= PyCF_TYPE_COMMENTS if isinstance(feature_version, tuple): major, minor = feature_version # Should be a 2-tuple. assert major == 3 feature_version = minor elif feature_version is None: feature_version = -1 # Else it should be an int giving the minor version for 3.x. > return compile(source, filename, mode, flags, _feature_version=feature_version) E File "", line 2 E (0:1) E ^ E SyntaxError: invalid syntax ../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError ____________________________________________________________________ TestMarkTokens.test_slices _____________________________________________________________________ self = def test_slices(self): # Make sure we don't fail on parsing slices of the form `foo[4:]`. source = "(foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::2, :], [a[:]][::-1])" > m = self.create_mark_checker(source) tests/test_mark_tokens.py:230: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:43: in create_mark_checker checker.verify_all_nodes(self) tests/tools.py:90: in verify_all_nodes rebuilt_node = test_case.parse_snippet(text, node) tests/test_mark_tokens.py:735: in parse_snippet return self.module.parse('_\n(' + text + ')').body[1].value _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ source = '_\n(:3)', filename = '', mode = 'exec' def parse(source, filename='', mode='exec', *, type_comments=False, feature_version=None): """ Parse the source into an AST node. Equivalent to compile(source, filename, mode, PyCF_ONLY_AST). Pass type_comments=True to get back type comments where the syntax allows. """ flags = PyCF_ONLY_AST if type_comments: flags |= PyCF_TYPE_COMMENTS if isinstance(feature_version, tuple): major, minor = feature_version # Should be a 2-tuple. assert major == 3 feature_version = minor elif feature_version is None: feature_version = -1 # Else it should be an int giving the minor version for 3.x. > return compile(source, filename, mode, flags, _feature_version=feature_version) E File "", line 2 E (:3) E ^ E SyntaxError: invalid syntax ../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError __________________________________________________________________ TestMarkTokens.test_sys_modules __________________________________________________________________ self = def test_sys_modules(self): """ Verify all nodes on source files obtained from sys.modules. This can take a long time as there are many modules, so it only tests all modules if the environment variable ASTTOKENS_SLOW_TESTS has been set. """ modules = list(sys.modules.values()) if not os.environ.get('ASTTOKENS_SLOW_TESTS'): modules = modules[:20] start = time() for module in modules: # Don't let this test (which runs twice) take longer than 13 minutes # to avoid the travis build time limit of 30 minutes if time() - start > 13 * 60: break try: filename = inspect.getsourcefile(module) except TypeError: continue if not filename: continue filename = os.path.abspath(filename) print(filename) try: with io.open(filename) as f: source = f.read() except OSError: continue # Astroid fails with a syntax error if a type comment is on its own line if self.is_astroid_test and re.search(r'^\s*# type: ', source, re.MULTILINE): print('Skipping', filename) continue > self.create_mark_checker(source) tests/test_mark_tokens.py:639: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/test_mark_tokens.py:32: in create_mark_checker atok = self.create_asttokens(source) tests/test_mark_tokens.py:48: in create_asttokens return ASTTokens(source, parse=True) asttokens/asttokens.py:65: in __init__ self.mark_tokens(self._tree) asttokens/asttokens.py:76: in mark_tokens MarkTokens(self).visit_tree(root_node) asttokens/mark_tokens.py:49: in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) asttokens/util.py:186: in visit_tree ret = postvisit(current, par_value, value) asttokens/mark_tokens.py:92: in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) asttokens/mark_tokens.py:334: in visit_keyword util.expect_token(name, token.NAME, node.arg) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ token = Token(type=1, string='message', start=(235, 12), end=(235, 19), line=" message = '# ' + message\n", index=1087, startpos=7481, endpos=7488) tok_type = 1, tok_str = 'file' def expect_token(token, tok_type, tok_str=None): """ Verifies that the given token is of the expected type. If tok_str is given, the token string is verified too. If the token doesn't match, raises an informative ValueError. """ if not match_token(token, tok_type, tok_str): > raise ValueError("Expected token %s, got %s on line %s col %s" % ( token_repr(tok_type, tok_str), str(token), token.start[0], token.start[1] + 1)) E ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13 asttokens/util.py:56: ValueError ----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------ /home/tcaswell/.pybuild/bleeding/lib/python3.9/importlib/_bootstrap.py ====================================================================== short test summary info ====================================================================== FAILED tests/test_astroid.py::TestAstroid::test_complex_slice_and_parens - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed: FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - AssertionError: 'Slice(\n lower=None,\n upper=None,\n step=None)' != "Name(name='d')" FAILED tests/test_astroid.py::TestAstroid::test_nonascii - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed: FAILED tests/test_astroid.py::TestAstroid::test_slices - AssertionError: 'Slice(\n lower=None,\n upper=Const(value=3),\n step=None)' != 'Const(value=3)' FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - AssertionError: "Slice(\n lower=None,\n upper=BinOp(\n[245 chars]one)" != "BinOp(\n op='-',\n ... FAILED tests/test_mark_tokens.py::TestMarkTokens::test_assignment_expressions - ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_complex_slice_and_parens - File "", line 2 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_decorators - ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture8 - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture9 - ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_keyword_arg_only - ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_mark_tokens_simple - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_nonascii - File "", line 2 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_slices - File "", line 2 FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules - ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13 ======================================================== 15 failed, 91 passed, 1 skipped, 1 warning in 6.23s ======================================================== ```

full IPython traceback

``` (bleeding) ✘ ~/source/other_source/asttokens [master {origin/master}|✔ ] jupiter@17:52 ➤ ipython Python 3.9.0a5+ (heads/master:799d7d61a9, Apr 6 2020, 17:38:49) Type 'copyright', 'credits' or 'license' for more information IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help. In [1]: 1/0 Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in 1/0 ZeroDivisionError: division by zero During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback stb = value._render_traceback_() AttributeError: 'ZeroDivisionError' object has no attribute '_render_traceback_' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing args = executing_cache[key] KeyError: (, 140371038733216, 142) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3254, in run_ast_nodes if (await self.run_code(code, result, async_=asy)): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3348, in run_code self.showtraceback(running_compiled_code=True) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback stb = self.InteractiveTB.structured_traceback(etype, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback return FormattedTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback return VerboseTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole records = self.get_records(etb, number_of_lines_of_context, tb_offset) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:] File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data yield from collapse_repeated( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated yield from map(mapper, original_group) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper return cls(f, options) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__ self.executing = Source.executing(frame_or_tb) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing source = cls.for_frame(frame) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {}) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename result = source_cache[filename] = cls(filename, lines) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__ self.asttokens() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens return ASTTokens( File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__ self.mark_tokens(self._tree) File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens MarkTokens(self).visit_tree(root_node) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree ret = postvisit(current, par_value, value) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword util.expect_token(name, token.NAME, node.arg) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token raise ValueError("Expected token %s, got %s on line %s col %s" % ( ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback stb = value._render_traceback_() AttributeError: 'ValueError' object has no attribute '_render_traceback_' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing args = executing_cache[key] KeyError: (, 140371038732864, 690) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2886, in _run_cell return runner(coro) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner coro.send(None) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3062, in run_cell_async has_raised = await self.run_ast_nodes(code_ast.body, cell_name, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3273, in run_ast_nodes self.showtraceback() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback stb = self.InteractiveTB.structured_traceback(etype, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback return FormattedTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback return VerboseTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole records = self.get_records(etb, number_of_lines_of_context, tb_offset) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:] File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data yield from collapse_repeated( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated yield from map(mapper, original_group) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper return cls(f, options) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__ self.executing = Source.executing(frame_or_tb) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing source = cls.for_frame(frame) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {}) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename result = source_cache[filename] = cls(filename, lines) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__ self.asttokens() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens return ASTTokens( File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__ self.mark_tokens(self._tree) File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens MarkTokens(self).visit_tree(root_node) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree ret = postvisit(current, par_value, value) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword util.expect_token(name, token.NAME, node.arg) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token raise ValueError("Expected token %s, got %s on line %s col %s" % ( ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback stb = value._render_traceback_() AttributeError: 'ValueError' object has no attribute '_render_traceback_' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing args = executing_cache[key] KeyError: (, 140371038702208, 60) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tcaswell/.virtualenvs/bleeding/bin/ipython", line 8, in sys.exit(start_ipython()) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/__init__.py", line 126, in start_ipython return launch_new_instance(argv=argv, **kwargs) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/traitlets/config/application.py", line 664, in launch_instance app.start() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/ipapp.py", line 356, in start self.shell.mainloop() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 559, in mainloop self.interact() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 550, in interact self.run_cell(code, store_history=True) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2857, in run_cell result = self._run_cell( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2891, in _run_cell self.showtraceback(running_compiled_code=True) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback stb = self.InteractiveTB.structured_traceback(etype, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback return FormattedTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback return VerboseTB.structured_traceback( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context, File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole records = self.get_records(etb, number_of_lines_of_context, tb_offset) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:] File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data yield from collapse_repeated( File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated yield from map(mapper, original_group) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper return cls(f, options) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__ self.executing = Source.executing(frame_or_tb) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing source = cls.for_frame(frame) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {}) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename result = source_cache[filename] = cls(filename, lines) File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__ self.asttokens() File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens return ASTTokens( File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__ self.mark_tokens(self._tree) File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens MarkTokens(self).visit_tree(root_node) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree util.visit_tree(node, self._visit_before_children, self._visit_after_children) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree ret = postvisit(current, par_value, value) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last) File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword util.expect_token(name, token.NAME, node.arg) File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token raise ValueError("Expected token %s, got %s on line %s col %s" % ( ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9 If you suspect this is an IPython 8.0.0.dev bug, please report it at: https://github.com/ipython/ipython/issues or send an email to the mailing list at ipython-dev@python.org You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it. Extra-detailed tracebacks for bug-reporting purposes can be enabled via: %config Application.verbose_crash=True ```

attn @alexmojaki @Carreau

alexmojaki commented 4 years ago

Thanks @tacaswell for the report!

OK the first problem I see is that slices are now expressions and location information has been added which has fixed behaviour in both asttokens and ast:

import ast

from asttokens import ASTTokens

source = """
x[1:2]
"""

tree = ast.parse(source)
print(ast.dump(tree))

node = tree.body[0].value.slice
print(ast.dump(node))
print(f"{ast.get_source_segment(source, node)=}")

atok = ASTTokens(source, tree=tree)
print(f"{atok.get_text(node)=}")

print(f"{ast.Slice.__base__=}")

In Python 3.8:

ast.get_source_segment(source, node)=None
atok.get_text(node)='2'
ast.Slice.__base__=<class '_ast.slice'>

(the exact result of atok.get_text(node) varies quite a bit with different slices)

In 3.9:

ast.get_source_segment(source, node)=':2'
atok.get_text(node)=':2'
ast.Slice.__base__=<class 'ast.expr'>

Because they're now expressions, the tests try to verify the slice nodes and parse source code like ':2' which of course is a syntax error.

@dsagal how should we handle this? By far the easiest course is to just ignore slices in tests, as has done before.

alexmojaki commented 4 years ago

Oh and ExtSlice is now represented by Tuple. See https://bugs.python.org/issue34822

alexmojaki commented 4 years ago

Overall, there are 5 astroid failures and 10 regular failures.

If I ignore slices in verify_all_nodes, then there are 0 astroid failures and 7 regular failures.

The other problem is that keyword nodes have changed: https://bugs.python.org/issue40141

This simple script fails:

import asttokens

asttokens.ASTTokens("""
foo(x=y)
""", parse=1)

If in addition to ignoring slices I remove MarkTokens.visit_keyword, then there are 2 astroid failures and 0 regular failures!

That's all I have time for now, will look at this again later.

dsagal commented 4 years ago

Thank you @alexmojaki !

tacaswell commented 4 years ago

Awesome! Thank you for the quick fix (even if I only just noticed today)!

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.