listmaker is not part of Python3 grammar

[fspeech]

Python3 support is broken right now over symbol.listmaker http://python.readthedocs.org/en/latest/reference/grammar.html

These are the kind of things that could bite over and over again due to the cst implementation (vs ast).

[portante]

Are the AST trees the same between Python3 and Python2?

[fspeech]

Unfortunately it seems the answer is no, but it does seem the change is directly related to the change in semantics in the language definition.

Here is the SVN repo for Python.asdl: http://hg.python.org/cpython/file/3f739f42be51/Parser/Python.asdl

I don't know how else to get the diff history other than through the "file log", but some times you have to dig to find the asdl change as it is bundled with other file update logs as well.

For example the most recent meaningful change is related to "yield from":

 3.1 --- a/Parser/Python.asdl
 3.2 +++ b/Parser/Python.asdl
 3.3 @@ -59,7 +59,8 @@ module Python
 3.4         | DictComp(expr key, expr value, comprehension* generators)
 3.5         | GeneratorExp(expr elt, comprehension* generators)
 3.6         -- the grammar constrains where yield expressions can occur
 3.7 -       | Yield(int is_from, expr? value)
 3.8 +       | Yield(expr? value)
 3.9 +             | YieldFrom(expr? value)
3.10         -- need sequences for compare to distinguish between
3.11         -- x < 4 < 3 and (x < 4) < 3
3.12         | Compare(expr left, cmpop* ops, expr* comparators)

It is dated from 8 months ago. The diff is buried with a bunch of changes in the other files. The next one is from 13 months ago relates to Bytes:

   2.1 --- a/Parser/Python.asdl
 2.2 +++ b/Parser/Python.asdl
 2.3 @@ -1,4 +1,4 @@
 2.4 --- ASDL's four builtin types are identifier, int, string, object
 2.5 +-- ASDL's five builtin types are identifier, int, string, bytes, object
 2.6  
 2.7  module Python
 2.8  {
 2.9 @@ -67,7 +67,7 @@ module Python
2.10             expr? starargs, expr? kwargs)
2.11         | Num(object n) -- a number as a PyObject.
2.12         | Str(string s) -- need to specify raw, unicode, etc?
2.13 -       | Bytes(string s)
2.14 +       | Bytes(bytes s)
2.15         | Ellipsis
2.16         -- other literals? bools?
2.17  

On Sun, Sep 30, 2012 at 8:42 PM, Peter Portante notifications@github.comwrote:

Are the AST trees the same between Python3 and Python2?

— Reply to this email directly or view it on GitHubhttps://github.com/portante/pycscope/issues/12#issuecomment-9021247.

[fspeech]

Okay I worked out the difference with p3.3 with p2.7 cumulative diffs based on the python ref docs. (Had to hand format a bit to line up the diffs). I think the key thing is there are two type of changes: 1) Things that are not backwards compatible: for example Raise/Try have become Raise/TryExcept/TryFinally 2) Things that are backwards compatible. For example FunctionDef now includes returns stmts as well. But this is backwards compatible. If you only use p2.7 ast children, it will work the same in p3.3.

--- ../py/p2ast 2012-10-01 09:14:57 -0700
+++ ../py/p3ast 2012-09-30 22:51:24 -0700
@@ -1,6 +1,6 @@
--- ASDL's five builtin types are identifier, int, string, object, bool
+-- ASDL's five builtin types are identifier, int, string, bytes, object
-module Python version "$Revision$"
+module Python
 {
  mod = Module(stmt* body)
     | Interactive(stmt* body)
@@ -10,9 +10,12 @@
     | Suite(stmt* body)
  stmt = FunctionDef(identifier name, arguments args,
-                            stmt* body, expr* decorator_list)
+                           stmt* body, expr* decorator_list, expr? returns)
       | ClassDef(identifier name,
              expr* bases,
+             keyword* keywords,
+             expr? starargs,
+             expr? kwargs,
              stmt* body,
              expr* decorator_list)
       | Return(expr? value)
@@ -21,30 +24,21 @@
       | Assign(expr* targets, expr value)
       | AugAssign(expr target, operator op, expr value)
-      -- not sure if bool is allowed, can always use int
-      | Print(expr? dest, expr* values, bool nl)
-
       -- use 'orelse' because else is a keyword in target languages
       | For(expr target, expr iter, stmt* body, stmt* orelse)
       | While(expr test, stmt* body, stmt* orelse)
       | If(expr test, stmt* body, stmt* orelse)
-      | With(expr context_expr, expr? optional_vars, stmt* body)
+          | With(withitem* items, stmt* body)
-      -- 'type' is a bad name
-      | Raise(expr? type, expr? inst, expr? tback)
-      | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse)
-      | TryFinally(stmt* body, stmt* finalbody)
+          | Raise(expr? exc, expr? cause)
+          | Try(stmt* body, excepthandler* handlers, stmt* orelse,
stmt* finalbody)
       | Assert(expr test, expr? msg)
       | Import(alias* names)
       | ImportFrom(identifier? module, alias* names, int? level)
-      -- Doesn't capture requirement that locals must be
-      -- defined if globals is
-      -- still supports use as a function!
-      | Exec(expr body, expr? globals, expr? locals)
-
       | Global(identifier* names)
+          | Nonlocal(identifier* names)
       | Expr(expr value)
       | Pass | Break | Continue
@@ -66,19 +60,22 @@
      | GeneratorExp(expr elt, comprehension* generators)
      -- the grammar constrains where yield expressions can occur
      | Yield(expr? value)
+             | YieldFrom(expr? value)
      -- need sequences for compare to distinguish between
      -- x < 4 < 3 and (x < 4) < 3
      | Compare(expr left, cmpop* ops, expr* comparators)
      | Call(expr func, expr* args, keyword* keywords,
  expr? starargs, expr? kwargs)
-     | Repr(expr value)
      | Num(object n) -- a number as a PyObject.
      | Str(string s) -- need to specify raw, unicode, etc?
+         | Bytes(bytes s)
+         | Ellipsis
      -- other literals? bools?
      -- the following expression can appear in assignment context
      | Attribute(expr value, identifier attr, expr_context ctx)
      | Subscript(expr value, slice slice, expr_context ctx)
+         | Starred(expr value, expr_context ctx)
      | Name(identifier id, expr_context ctx)
      | List(expr* elts, expr_context ctx)
      | Tuple(expr* elts, expr_context ctx)
@@ -88,7 +85,7 @@
  expr_context = Load | Store | Del | AugLoad | AugStore | Param
- slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step)
+    slice = Slice(expr? lower, expr? upper, expr? step)
       | ExtSlice(slice* dims)
       | Index(expr value)
@@ -103,17 +100,21 @@
  comprehension = (expr target, expr iter, expr* ifs)
- -- not sure what to call the first argument for raise and except
- excepthandler = ExceptHandler(expr? type, expr? name, stmt* body)
+    excepthandler = ExceptHandler(expr? type, identifier? name, stmt* body)
                 attributes (int lineno, int col_offset)
- arguments = (expr* args, identifier? vararg,
-     identifier? kwarg, expr* defaults)
+    arguments = (arg* args, identifier? vararg, expr? varargannotation,
+                     arg* kwonlyargs, identifier? kwarg,
+                     expr? kwargannotation, expr* defaults,
+                     expr* kw_defaults)
+    arg = (identifier arg, expr? annotation)
         -- keyword arguments supplied to call
         keyword = (identifier arg, expr value)
         -- import name with optional 'as' alias.
         alias = (identifier name, identifier? asname)
+
+    withitem = (expr context_expr, expr? optional_vars)
 }

On Sun, Sep 30, 2012 at 8:42 PM, Peter Portante notifications@github.com wrote:

Are the AST trees the same between Python3 and Python2?

— Reply to this email directly or view it on GitHub.

[portante]

Thanks for the report and the comments about AST vs. CST. I have opened a separate issue to track such work, since the fix for this issue is rather trivial (thankfully).

Please check this out and let me know if it still does not work for you, reopen if it does not.