search

bukzor / RefactorLib

A library for refactorization. (cheetah and python, currently)
5 stars 2 forks source link

python: fail to parse when list comprehension ends with a comma #29

Closed asottile closed 4 years ago

asottile commented 8 years ago 
y = lambda x: x
y(
    t for t in (1, 2, 3),
)
$ xmlfrom !$
xmlfrom test.py
Traceback (most recent call last):
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/bin/xmlfrom", line 9, in <module>
    load_entry_point('refactorlib==0.11.0', 'console_scripts', 'xmlfrom')()
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/cli/xmlfrom.py", line 25, in cli
    return main(argv, stdout)
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/cli/xmlfrom.py", line 19, in main
    stdout.write(xmlfrom(argv[1]))
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/cli/xmlfrom.py", line 11, in xmlfrom
    tree = parse(filename).getroottree()
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/parse.py", line 18, in parse
    return filetype.parser(source, encoding)
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/python/parse.py", line 28, in parse
    lib2to3_python = lib2to3_parse(python_contents)
  File "/nail/home/asottile/pg/yelp-main/virtualenv_run/lib/python2.7/site-packages/refactorlib/python/parse.py", line 53, in lib2to3_parse
    tree = py2_driver.parse_string(python_contents, True)
  File "/usr/lib64/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib64/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib64/python2.7/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=8, value=u')', context=(u'\n', (4, 0))
asottile commented 6 years ago

fun times, this is a syntax error in python3.7+

bukzor commented 6 years ago

FFS.

bukzor commented 6 years ago

They really need to add :sob: to the reaction list.

asottile commented 6 years ago

but actually 😢 😭 😢

asottile commented 6 years ago

here's how I noticed :salt: https://github.com/asottile/pyupgrade/pull/45/commits/6a1816aabf2d0dc57f7a5af55258f803bbb3bf06

asottile commented 4 years ago

instead of fixing lib2to3, they changed how the parser works so this is now kinda "invalid"

$ python3.6 -c 'print(1 for _ in range(4),)'
<generator object <genexpr> at 0x7fc713b9baf0>
$ python3.7 -c 'print(1 for _ in range(4),)'
  File "<string>", line 1
SyntaxError: Generator expression must be parenthesized