Test failure in Python 3.6.3

[felixonmars]

Looks like a test-only failure, though.

======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor-0.6/tests/test_rtrip.py", line 24, in test_convert_stdlib
    self.assertEqual(result, [])
AssertionError: Lists differ: ['/usr/lib/python3.6/test/test_fstring.py'[34 chars].py'] != []

First list contains 2 additional elements.
First extra element 0:
'/usr/lib/python3.6/test/test_fstring.py'

+ []
- ['/usr/lib/python3.6/test/test_fstring.py',
-  '/usr/lib/python3.6/idlelib/grep.py']

[berkerpeksag]

Thanks for the report. I think the cause of both failures is f-strings. We probably miss some cases when we implement it. Also, did you get any output from astor.rtrip? It should normally prints out broken nodes: https://github.com/berkerpeksag/astor/blob/68fbf65ed4eb7a9d5b0a3a380b57d3329012c2a4/astor/rtrip.py#L150-L151

[felixonmars]

Hrm, didn't see that.

root: INFO: Converting /usr/lib/python3.6/idlelib/grep.py
root: WARNING:     calculating dump -- bad

root: INFO: Converting /usr/lib/python3.6/test/test_fstring.py
root: WARNING:     calculating dump -- bad

Files failed to round-trip to AST:
root: WARNING:     /usr/lib/python3.6/test/test_fstring.py
root: WARNING:     /usr/lib/python3.6/idlelib/grep.py

That's all I found besides other INFO entries.

[berkerpeksag]

Thanks! I'll check what's wrong with 3.6.3 this weekend.

[radomirbosak]

How do I reproduce this? For me all tests pass

$ python3.6 -m nose -v
test_annassign (tests.test_code_gen.CodegenTestCase) ... ok
[...]
test_codegen_from_root (tests.test_misc.PublicAPITestCase) ... ok
test_convert_stdlib (tests.test_rtrip.RtripTestCase) ... ok

----------------------------------------------------------------------
Ran 34 tests in 18.100s

OK

The same happens when running via tox:

$ tox -e py36
py36 develop-inst-noop: /home/lukoa/repos/astor
py36 installed: -e git+git@github.com:radomirbosak/astor.git@b47718fa095e456c064d3d222f296fccfe36266b#egg=astor,nose==1.3.7
py36 runtests: PYTHONHASHSEED='2587239569'
py36 runtests: commands[0] | nosetests -v --nocapture
test_annassign (tests.test_code_gen.CodegenTestCase) ... ok
[...]
test_codegen_from_root (tests.test_misc.PublicAPITestCase) ... ok
test_convert_stdlib (tests.test_rtrip.RtripTestCase) ... ok

----------------------------------------------------------------------
Ran 34 tests in 16.442s

OK
__________________________________ summary ___________________________________
  py36: commands succeeded
  congratulations :)

[berkerpeksag]

@radomirbosak, thank you for trying to reproduce this! Do you have the latest 3.6 release? (If you have 3.6.5 installed with stdlib tests and IDLE installed, I think we can close this as outdated)

Note that almost all distros don't include stdlib tests and don't install IDLE by default. So you may want to check whether test/test_fstring.py and idlelib/grep.py tested by test_convert_stdlib.

[felixonmars]

I'm still getting this in 3.6.5 - besides one more missing item:

======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor-0.6.2/tests/test_rtrip.py", line 24, in test_convert_stdlib
    self.assertEqual(result, [])
AssertionError: Lists differ: ['/usr/lib/python3.6/netrc.py', '/usr/lib/[65 chars].py'] != []

First list contains 3 additional elements.
First extra element 0:
'/usr/lib/python3.6/netrc.py'

+ []
- ['/usr/lib/python3.6/netrc.py',
-  '/usr/lib/python3.6/test/test_fstring.py',
-  '/usr/lib/python3.6/idlelib/grep.py']

Files failed to round-trip to AST:
root: WARNING:     /usr/lib/python3.6/netrc.py
root: WARNING:     /usr/lib/python3.6/test/test_fstring.py
root: WARNING:     /usr/lib/python3.6/idlelib/grep.py

[berkerpeksag]

The only change that we made to Lib/netrc.py in the 3.6 branch is https://github.com/python/cpython/commit/5fbe5e161c969bc8a0d44a301152f8bf5afe0fc7.

I think there is an escaping issue here. Here's a simple test case:

def test_issue_89(self):
    source = """
    x = f'{host}\\n\\t{port}\\n'
    """
    self.assertSrcRoundtripsGtVer(source, (3, 6))

Test result:

======================================================================
FAIL: test_issue_89 (tests.test_code_gen.CodegenTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/berker/projects/astor/tests/test_code_gen.py", line 414, in test_issue_89
    self.assertSrcRoundtripsGtVer(source, (3, 6))
  File "/home/berker/projects/astor/tests/test_code_gen.py", line 74, in assertSrcRoundtripsGtVer
    self.assertSrcRoundtrips(source)
  File "/home/berker/projects/astor/tests/test_code_gen.py", line 62, in assertSrcRoundtrips
    self.assertEqual(self.to_source(ast.parse(srctxt)).rstrip(), srctxt)
AssertionError: "x = \nf'{host}\\n\\t{port}'" != "x = f'{host}\\n\\t{port}\\n'"

Also, if I drop the trailing \n, the test will pass:

source = """
x = f'{host}\\n\\t{port}'
"""

@radomirbosak would you like to work on this?

[radomirbosak]

@berkerpeksag I can take a look.

After installing the stdlib tests and idle packages I was able to reproduce the issue (on python 3.6.5).