Open clbarnes opened 5 years ago
I think this is a good idea in general, but to me this feels like actually a big project. Does it need to be in PyO3 itself, or could it be done as a third-party extension?
It could certainly be a 3rd-party extension. I'm afraid I don't know enough about pyo3's internals even to judge how big a job it would be - it seemed to me like pyo3 knows which types can interconvert, and rust knows what types its functions support, so it would be a case of formatting existing data rather than trying to generate it in the first place, but I fully accept that that's probably wrong.
Sounds interesting, thanks.
But I have to learn .pyi
format first :thinking:
I've already done a bit of research on the topic, which resulted in #305 and #310.
By default, there are tools that can open native (and pure python) modules and generate stubs from them. Pycharm for example automatically generates stubs for all installed modules; I've added the stub for the word-count example below. Python 3 has additionally introduced __annotations__
, which is a dict containing type annotations. However python doesn't allow to set __annotations__
for native functions, so we need to come up with something different. (Another thing to note that's not in the issues linked above is that PEP 563 will change __annotations__
to a Dict[str, str]
, so that annotations need to be resolved at runtime. However typing.get_type_hints
will still work so that shouldn't be a big problem for most tools.)
A simple way to get at least the names and positions of the arguments is generating a text signature (#310), as e.g. orjson currently does manually. I think that that should be done first.
If we want more advanced annotations, we'd need to create our own metadata format. See e.g. wasm-bingen's approach with WasmDescribe::describe()
. We need to write the type-info at compile to some constant/field, while writing stub files should be done by an external tool, in our case setuptools-rust or pyo3-pack. wasm-pack has a cli tool and wasm-pack for that
But I have to learn .pyi format first thinking
They are just normal python files with the implementation replaced by ...
.
# encoding: utf-8
# module word_count.word_count
# from /home/konsti/pyo3/.venv/lib/python3.6/site-packages/word_count/word_count.cpython-36m-x86_64-linux-gnu.so
# by generator 1.147
# no doc
# no imports
# functions
def count_line(*args, **kwargs): # real signature unknown
""" Count the occurences of needle in line, case insensitive """
pass
# classes
class WordCounter(object):
""" Represents a file that can be searched """
def search(self, *args, **kwargs): # real signature unknown
""" Searches for the word, parallelized by rayon """
pass
def search_sequential(self, *args, **kwargs): # real signature unknown
""" Searches for a word in a classic sequential fashion """
pass
def __init__(self, *args, **kwargs): # real signature unknown
pass
@staticmethod # known case of __new__
def __new__(*args, **kwargs): # real signature unknown
""" Create and return a new object. See help(type) for accurate signature. """
pass
# variables with complex values
__all__ = [
'__doc__',
'count_line',
'WordCounter',
]
__loader__ = None # (!) real value is '<_frozen_importlib_external.ExtensionFileLoader object at 0x7f89093d9940>'
__spec__ = None # (!) real value is "ModuleSpec(name='word_count.word_count', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f89093d9940>, origin='/home/konsti/pyo3/.venv/lib/python3.6/site-packages/word_count/word_count.cpython-36m-x86_64-linux-gnu.so')"
hey, thanks for working on this library. I've been looking around about this, so here are my notes, which you might find useful
Looks like mypy solves it by reusing the python parser module to re-parse each module and create an AST. Then it creates Stubs for AST nodes.
https://github.com/python/mypy/blob/master/mypy/stubgen.py
It might be too big a problem to solve inside pyo3, at least mypy shows how one could do it.
Yeah but we need here is just type hints for pymethods
, right?
by manually adding the signatures to the docstrings, pyls can autocomplete all the methods. See https://github.com/PyO3/pyo3/issues/310#issuecomment-556038749 for a proposal
Does that include type annotations or is that too much to ask? :wink:
My only concern is that if we're writing explicit header stubs anyway, we may as well just write a full pyi stub file and at least gain access to the tooling around that.
Does that include type annotations or is that too much to ask?
From what I understand, type annotations don't work at all for return types (CPython doesn't recognize the signature and put it in __text_signature__
) and sort-of works for arguments (__text_signature__
gets set properly but inspect.signature
raises an exception).
My only concern is that if we're writing explicit header stubs anyway, we may as well just write a full pyi stub file and at least gain access to the tooling around that.
I think having signatures is useful for those cases when you need help()
from Python and/or you want inspect.signature
to work (don't think .pyi
files are used for either, though could be wrong). That should not exclude generating .pyi
files in addition, which would also be nice.
I'd also find this feature really useful.
Ideally what I'd like is that PyO3 produces a .pyi
stub with type information both for parameters and return types, and then strips this information out when producing the .pyd
file so that the docstring that's included in the .pyd
file meets the constraints of the CPython implementation (e.g. no return type annotation). I don't think it's a huge amount of work - tokenising the text_signature
string to remove the trailing -> MyType
before handling it like we do now, and templating some text to produce the stub. Given the produced stub is just for developer convenience, it doesn't have to be watertight on the first iteration.
Here is how napi-rs does typescript .d.ts
file generation for inspiration: https://github.com/napi-rs/napi-rs/pull/696/commits/741dd93aff5a50954fca45ba3c6a84a6d562fc8f
In case someone else finds this thread, the tracking issue is https://github.com/PyO3/pyo3/issues/2454.
Hi, is there any updates? Thanks
Hello, @fzyzcjy and @Haaroon. Thank you for your interest in this issue. All updates related to this issue have been linked to https://github.com/PyO3/pyo3/issues/2454. Kindly refrain from requesting updates through comments.
CC maintainers: please hide this comment along with the previous two to reduce noise for new comers.
Not sure how possible this would be, but it would be awesome if we could give our IDEs an idea of what's going on inside a pyo3 function/class with
.pyi
stub files. Pyo3 already knows what rust and python types can interconvert: expressing this information for the pyo3 objects in a stub file would really help to bridge the gap.