search

google / pasta

Library to refactor python code through AST manipulation.
Apache License 2.0
341 stars 40 forks source link

Failed to recognize nested try: #23

Closed williamjamir closed 6 years ago

williamjamir commented 6 years ago

Hi,

Running the parser against a file with "nested try" results in the following error:

pasta.base.annotate.AnnotationError: Expected 'error' but found 'try'
line 2:     try:

I found this issue with 

Python 2.7.8 :: Continuum Analytics, Inc.
Python 2.7.9 :: Continuum Analytics, Inc.
Python 2.7.12

(In python 3.4.5 works just fine)

Test.py

try:
    try:
        error = 1/0
    except:
        print('1')
finally:
    print('2')

Script.py:

import os
import pasta

path = r'test.py'
with open(path) as file:
   tree = pasta.parse(file.read())

Full Log:

Traceback (most recent call last):
  File "script.py", line 10, in <module>
    tree = pasta.parse(file.read())
  File "w:\william\repo_pasta\pasta\__init__.py", line 25, in parse
    annotator.visit(t)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 168, in visit_Module
    self.generic_visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 249, in generic_visit
    self.visit(item)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 367, in visit_TryFinally
    self.visit(stmt)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 379, in visit_TryExcept
    self.visit(stmt)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 470, in visit_Assign
    self.visit(target)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1048, in visit
    super(AstAnnotator, self).visit(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "W:\alfasim\envs\test_pasta\lib\ast.py", line 241, in visit
    return visitor(node)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 747, in visit_Name
    self.token(node.id)
  File "w:\william\repo_pasta\pasta\base\annotate.py", line 1106, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'error' but found 'try'
line 2:     try:
soupytwist commented 6 years ago

Hmm, actually it seems this is an issue in 2.7.6 as well. Thanks for reporting!

soupytwist commented 6 years ago

Aye, this is a difficult case because these two parse to the same ast:

try:
  try:
    a
  except:
    b
finally:
  c

and

try:
  a
except:
  b
finally:
  c

Both result in: "Module(body=[TryFinally(body=[TryExcept(body=[Expr(value=Name(id='a', ctx=Load()))], handlers=[ExceptHandler(type=None, name=None, body=[Expr(value=Name(id='b', ctx=Load()))])], orelse=[])], finalbody=[Expr(value=Name(id='c', ctx=Load()))])])"

The extra desugaring throws off the annotator.

akov commented 6 years ago

Hmm should we commit to supporting cases like this?

It is a bit weird to respect it since the ast doesn't really respect this pattern (which I assume is because these two forms are semantically equivalent anyway).

williamjamir commented 6 years ago

Hi @soupytwist

I tested here your last commit with the following file and everything was ok:

try:
    print('1')
    try:
        error = 1/0
    except:
        print('2')
finally:
    print('3')

But, when I commented the print('1') the parser could not recognize the nested try

file: test.py

try:
    # print('1')
    try:
        error = 1/0
    except:
        print('2')
finally:
    print('3')

Running the file to show that it's a valid syntax =)

(pasta_py27) william@laptop:/opt/test_pasta$ python --version
Python 2.7.14
(pasta_py27) william@laptop:/opt/test_pasta$ python test.py 
2
3

Running the parse on this test.py file:

(pasta_py27) william@laptop:/opt/test_pasta$ python script.py 
Traceback (most recent call last):
  File "script.py", line 8, in <module>
    tree = pasta.parse(file.read())
  File "/opt/pasta/pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 168, in visit_Module
    self.generic_visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 373, in visit_TryFinally
    self.visit(stmt)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 79, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 386, in visit_TryExcept
    self.visit(stmt)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 477, in visit_Assign
    self.visit(target)
  File "/opt/pasta/pasta/base/annotate.py", line 1055, in visit
    super(AstAnnotator, self).visit(node)
  File "/opt/pasta/pasta/base/annotate.py", line 115, in visit
    super(BaseVisitor, self).visit(node)
  File "/home/william/miniconda3/envs/google_pasta_py27/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/opt/pasta/pasta/base/annotate.py", line 43, in wrapped
    f(self, node, *args, **kwargs)
  File "/opt/pasta/pasta/base/annotate.py", line 754, in visit_Name
    self.token(node.id)
  File "/opt/pasta/pasta/base/annotate.py", line 1118, in token
    token_val, token.src, token.start[0], token.line))
pasta.base.annotate.AnnotationError: Expected 'error' but found 'try'
line 3:     try:

cc @nicoddemus

soupytwist commented 6 years ago

re: @akov

It's legal python, and by principle #1, we need to be able to output exactly as it is read in. I plan to support it.

@williamjamir Thanks for the follow-up! Will work on fixing this.