search

bblfsh / python-driver

GNU General Public License v3.0
10 stars 16 forks source link

Errors while extracting all commits in pallets/jinja #143

Closed vmarkovtsev closed 5 years ago

vmarkovtsev commented 6 years ago

Hercules extracted UASTs from all the changed Python files in the history of https://github.com/pallets/jinja These are the errors. Unfortunately, the commit hashes were not recorded, but it can still be useful.

 349 / 1272 [===============>-------------------------------------------] 13m19s
setup.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 139, in process_request
    '\n------ Python3 errors:\n%s' % codeinfo['py3_ast_errors']
Exception: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 135\n    entry_points="""\n               ^\nSyntaxError: invalid syntax\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/usr/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 86\n    except CCompilerError, x:\n                         ^\nSyntaxError: invalid syntax\n']
 368 / 1272 [================>------------------------------------------] 16m12s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 376 / 1272 [=================>-----------------------------------------] 11m44s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 392 / 1272 [==================>-----------------------------------------] 8m34s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 394 / 1272 [==================>-----------------------------------------] 6m21s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 396 / 1272 [==================>-----------------------------------------] 3m51s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 614 / 1272 [============================>-------------------------------] 1m51s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
 826 / 1272 [=======================================>---------------------] 4m1s
docs/conf.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 144, in process_request
    ast = AstImprover(code, orig_ast).parse()
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 425, in parse
    res = self.visit(self._astdict, root=True)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in visit_other_field
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 528, in <listcomp>
    return [self.visit(x) for x in node]
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 440, in visit
    visit_result = meth(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 521, in visit_other
    node[field] = meth(node[field])
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 526, in visit_other_field
    return self.visit(node)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 442, in visit
    self.pos_sync.sync_node_pos(visit_result)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 149, in sync_node_pos
    token = self._pop_token(node_line, node_token)
  File "/usr/lib/python3.6/site-packages/python_driver/astimprove.py", line 112, in _pop_token
    line_value = literal_eval(t.value)
  File "/usr/lib/python3.6/ast.py", line 48, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 8
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 1-2: truncated \uXXXX escape
1089 / 1272 [===================================================>---------] 1m3s
jinja2/filters.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 139, in process_request
    '\n------ Python3 errors:\n%s' % codeinfo['py3_ast_errors']
Exception: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 989\n    \'urlize\':               do_urlize,\n           ^\nSyntaxError: invalid syntax\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/usr/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 989\n    \'urlize\':               do_urlize,\n           ^\nSyntaxError: invalid syntax\n']
1092 / 1272 [===================================================>---------]  34s
jinja2/filters.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 139, in process_request
    '\n------ Python3 errors:\n%s' % codeinfo['py3_ast_errors']
Exception: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 992\n    \'urlize\':               do_urlize,\n           ^\nSyntaxError: invalid syntax\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/usr/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 992\n    \'urlize\':               do_urlize,\n           ^\nSyntaxError: invalid syntax\n']
1236 / 1272 [==========================================================>--]  14s
setup.py: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/python_driver/requestprocessor.py", line 139, in process_request
    '\n------ Python3 errors:\n%s' % codeinfo['py3_ast_errors']
Exception: Errors produced trying to get an AST for both Python versions
------ Python2 errors:
[b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python2.7/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python2.7/ast.py", line 37, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 43\n    <<<<<<< HEAD\n     ^\nSyntaxError: invalid syntax\n']
------ Python3 errors:
['Traceback (most recent call last):\n  File "/usr/lib/python3.6/site-packages/pydetector/ast_checks.py", line 53, in check_ast\n    current_ast = ast2dict(code)\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 19, in ast2dict\n    return visitor.parse()\n  File "/usr/lib/python3.6/site-packages/pydetector/ast2dict.py", line 45, in parse\n    tree = ast.parse(self.codestr, mode=\'exec\')\n  File "/usr/lib/python3.6/ast.py", line 35, in parse\n    return compile(source, filename, mode, PyCF_ONLY_AST)\n  File "<unknown>", line 43\n    <<<<<<< HEAD\n     ^\nSyntaxError: invalid syntax\n']
1246 / 1272 [===========================================================>-]  20s
juanjux commented 6 years ago

Those are parsing errors from the native AST (invalid syntax), I'll take a closer look when I'm back but look like some of those files are broken (or not Python files).

juanjux commented 5 years ago

@vmarkovtsev I've successfully parsed all the Python files in that repo, with both the Python and Go clients (results attached below), could you confirm if this still happens? If that's the case, it could be a case of enry mis-identifing files as Python ones or maybe some already fixed bug in bblfshd or the sdk. In any case, it would help to know what are the files that are failing (I grepped some of the output when there is a traceback but the files I found I tried worked).

jinja_output.gz

juanjux commented 5 years ago

Temptatively closing, please reopen if you can reproduce.