A couple of _ast related issues

isidentical commented 3 years ago

I was checking out the type stub, and found these points;

There are a bunch of docstring: str nodes guarded with sys.version >= (3, 7). This is not actually true since python/cpython#7121 reverts the change that introduced docstring (that PR also targets 3.7). They can be safely eliminated
There is no type_comment on FunctionDef/AsyncFunctionDef (and maybe on other nodes)
string fields in the ASDL are encoded as str. The problem is string corresponds to typing.AnyStr, not str. (https://github.com/python/cpython/blob/cda99b4022daa08ac74b0420e9903cce883d91c6/Python/Python-ast.c#L1195-L1202)
As mentioned in this TODO:, every node has current position information (lineno/col_offset) however this is not the case.

While examining this, I got an idea of autogenerating stub files by inputting a program with ASDLs (abstract syntax description language). So I wrote generators/typing_stub.py to my ASDL implementation, also generated test stub file.

I didn't test using that stub file, so don't know if there is a problem with it, but from a high level view they look similar with the _ast.pyi here, plus it covers the points I just mentioned (be aware, it is completely auto generated). There is only a single problem with that stub file, it ignores one of the broken parts of our AST. Dict(expr* keys, expr* values) actually doesn't describe the Dict well since when one of the items is **star_unpack the keys will contain None (so it should be List[Optional[expr]] not List[expr]). The currently type stub handles it, so if anyone wants to make a PR by copying things over the auto generated type stub, this point should not be actually copied.

isidentical commented 3 years ago

Another thing I noticed is;

import ast
node = ast.AST(body=[])
print(node.body)

 $ mypy t.py                    
t.py:4: error: "AST" has no attribute "body"
Found 1 error in 1 file (checked 1 source file)

I believe (did not test) can be fixed using something similiar to types.SimpleNamespace. https://github.com/isidentical/pyasdl/blob/c44b9a87ae9b6cf5ab98ef0f32b1757964d5a0de/generators/PythonAST.pyi#L10-L12

srittau commented 3 years ago

Thank you and sorry for the late reply. Autogenerating the stub file sounds interesting indeed. I will take a closer look when I find the time.

srittau commented 3 years ago

The only problem I see with autogenerating is supporting multiple Python version, which we need to do in typeshed.

isidentical commented 3 years ago

The only problem I see with autogenerating is supporting multiple Python version, which we need to do in typeshed.

I don't quite get what you meant here, pyasdl's type stub generator generates for multiple versions. And also tries to do it in a gentle way. Check out the output stub.

srittau commented 3 years ago

I totally missed that, that looks great!

python / typeshed

A couple of _ast related issues #4760