search

kylebgorman / pynini

Read-only mirror of Pynini
http://pynini.opengrm.org
Apache License 2.0
118 stars 27 forks source link

docstrings usable by vscode? #46

Closed sbuser closed 3 years ago

sbuser commented 3 years ago

I've recently been introduced to FSTs and pynini via Nvidia's Nemo text processing repository. I found several issues in their grammars that I wanted to correct and having a lot of experience with regex, assumed I could provide some straightforward contributions.

Unfortunately, I must be denser than the average bear as I'm having a lot of trouble following how pynini is working even with the information on your website, your O'Reilly article, and several papers on the subject.

Part of the problem is that it took me a long time to try what you explicitly state in the "API Reference" section: help(pynini). I was relying on the docstrings via vscode and when I saw no help except typing hints, I assumed docstrings didn't exist.

I see now that help(pynini) provides what I was looking for and I've taken to copying the information there into the .pyi files that my IDE seems to be referencing to provide hints. Since these changes won't survive a reinstall, I'm wondering if these could be added here?

I assume this has to do with pynini being a cython extension or similar?

kylebgorman commented 3 years ago

Pynini ships with a complete set of type hints too, encoded as .pyi files, since release 2.1.1. If you're using that version or newer and VSCode still can't find them I would hypothesize this is a bug in VSCode since we follow PEP 561 (https://www.python.org/dev/peps/pep-0561/), the standard for distributing type hints.

On Thu, May 27, 2021 at 1:44 PM sbuser @.***> wrote:

I've recently been introduced to FSTs and pynini via Nvidia's Nemo text processing repository. I found several issues in their grammars that I wanted to correct and having a lot of experience with regex, assumed I could provide some straightforward contributions.

Unfortunately, I must be denser than the average bear as I'm having a lot of trouble following how pynini is working even with the information on your website, your O'Reilly article, and several papers on the subject.

Part of the problem is that it took me a long time to try what you explicitly state in the "API Reference" section: help(pynini). I was relying on the docstrings via vscode and when I saw no help except typing hints, I assumed docstrings didn't exist.

I see now that help(pynini) provides what I was looking for and I've taken to copying the information there into the .pyi files that my IDE seems to be referencing to provide hints. Since these changes won't survive a reinstall, I'm wondering if these could be added here?

I assume this has to do with pynini being a cython extension or similar?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kylebgorman/pynini/issues/46, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABG4ONOENW7XDT2TRBG2GLTP2AGRANCNFSM45UZAKSQ .

sbuser commented 3 years ago

The type hints are picked up but vscode wants to pull the docstrings from those same .pyi files (where they don't exist).

I see that the strings which appear in help(pynini) are in _pywrapfst.pyx, but they never make it into docstrings for the functions in the .pyi files. Is that basically working as intended?

In my case that means that I can tell from the IDE what types a accep() is expecting and providing, but have no clue what it's supposed to do. Following through to the function definition in vscode brings me to the .pyx version which is just a placeholder.

Others have figured it out so it's obviously doable, but when you're first looking at pynini-based code, in my view it's really hard to follow what's going on and accessible docstrings would help.

kylebgorman commented 3 years ago

What I expect to work is that I can see help(pywrapfst.VectorFst) or help(pynini.accep) from the interpreter, and static type tools like pytype and mypy can see the .pyi files. This fully works for me.

I don't know why VSCode would have issues with the docstrings, but I'd be happy to look at a PR if it's something I can correct on our end.

sbuser commented 3 years ago

Thanks Kyle. I'll see if I can put something together. It looks based on https://stackoverflow.com/questions/63179749/pycharm-and-vscode-not-picking-up-docstrings-from-cythonized-so and other posts like this is a known issue with cythonized modules. I'm so used to the docstrings just working that I never tried help().

sbuser commented 3 years ago

See also: https://github.com/microsoft/pylance-release/issues/674#issuecomment-738962588

The previous vscode python language server handled docstrings in cython extensions, the newer version (pylance) does not. So as you said, not your problem :D -- apologies.