Use mypy stub files to extract information about arguments and types of builtin and stdlib functions

[pylint-bot]

Originally reported by: BitBucket: ceridwenv, GitHub: @ceridwen

---

The MyPy people have done a lot of work creating files containing the signatures of builtin/stdlib functions at https://github.com/python/typeshed. By automatically processing the stub files, it would be possible to create more accurate mock ASTs for functions that aren't introspectable.

---

• Bitbucket: https://bitbucket.org/logilab/astroid/issue/254

[rogalski]

This one could be a huge feature and can effectively remove astroid.brain package. Does anyone know if there are any production-ready parsers for *.pyi file format?

[PCManticore]

I'm not sure, I think every tool has their own, e.g. mypy, pytype and PyCharm, but I might be wrong on this. It will be huge, but it won't replace astroid.brain entirely. astroid.brain is much more potent than the .pyi files, because we can modify the AST at will, which leads to advanced inference support for:

• builtins

• constructs that depends on highly dynamic code such as enums and namedtuples

• tailored support for various libraries

We can't have this out of the box with .pyi files, so brain is here to stay, although probably not in its current form.

[danilo-augusto]

What about using .pyi files as hints?

They could add initial typing information, which could be invalidated at the first available inference made by the brain module about the object, if any.

In the absence of any other information it can be the difference between being able to detect an issue or not.

For example, one can think of compiled components. Take a look at Pytorch. Most components are compiled, so it's unsafe code through the eyes of pylint/astroid. Also, the compiled modules are exposed through this line (auto-discover and globals() registering): https://github.com/pytorch/pytorch/blob/192a26249d8a2d50df0cbb7240c6013fd9958b9b/torch/__init__.py#L273

At this point, there seems to be little hope about any useful inference, right? We are about to give up. Most people will add torch.* to generated-members to silent pylint warnings about torch (which ironically often enough will represent a good chunk of logic coded in the project, remaining very opaque to introspections) and go on with life.

But it does not have to be this way: the file __init__.pyi (which comes from __init__.pyi.in, cf. [1]) is automatically populated with very rich descriptions of the exposed "API":

This amounts to (currently) 1,5K lines of rich information waiting to be leveraged. 😁

[1] https://github.com/pytorch/pytorch/blob/192a26249d8a2d50df0cbb7240c6013fd9958b9b/torch/__init__.pyi.in#L110