Improved docstring display support

[abgivant]

Environment data

VS Code version: 1.18 Python Extension version: 0.8.0 Python Version: 3.4 OS and version: Windows 10

There is currently very little support for python docstrings, at least using the reST format that we use in our project. Existing docstrings are displayed in the intellisense dialog, but the parameters aren't interpreted as parameters, meaning that there is no type inference being done off of the documentation.

The two main things that would be helpful are:

1. Recognize existing docstrings and defined parameters/types
2. Include the ability to auto-generate docstrings, inferring types from the function signature when possible

Thanks!

[MikhailArkhipov]

Related microsoft/vscode-python#62

[brettcannon]

Closing in preference for microsoft/vscode-python#62 since it has more participants.

[segevfiner]

This also applies to Google/NumPy style docstrings. See microsoft/vscode-python#675 which was closed in favor of tracking this in this issue.

[brettcannon]

While we should make sure 1. works by not collapsing the lines into a single sentence, 2. is handled by https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring .

[jxramos]

I'm curious if VSCode makes in any way this docstring display format configurable by the user. I experienced a change in the formatting style that I haven't yet been able to track down like seen below. I want to get back to the BEFORE state which was the default I'm pretty sure when I setup VSCode. What governs the rendering precisely that could cause such a change in appearance?

BEFORE

AFTER

[segevfiner]

The docstring display was rewritten at some point due to giving bad output for some cases. It probably still doesn't support rendering everything nicely and needs to be extended to support rendering more things.

(It convers the reStructuredText to Markdown for display in VS Code)

בתאריך 23 במרץ 2018 03:37,‏ "jxramos" notifications@github.com כתב:

I'm curious if VSCode makes in any way this docstring display format configurable by the user. I experienced a change in the formatting style that I haven't yet been able to track down like seen below. I want to get back to the BEFORE state which was the default I'm pretty sure when I setup VSCode. What governs the rendering precisely that could cause such a change in appearance?

BEFORE [image: defaultdocstringformatting] https://user-images.githubusercontent.com/13857038/37805406-4c1bdf36-2df7-11e8-80b2-7064b9ac0a7b.png

AFTER [image: changeddocstringformatter] https://user-images.githubusercontent.com/13857038/37805411-525e11b6-2df7-11e8-9c25-ab2395230b86.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vscode-python/issues/38#issuecomment-375502223, or mute the thread https://github.com/notifications/unsubscribe-auth/AXlg_-AVnWAKml1eAjzJkCh-pSXELS1Xks5thEOwgaJpZM4QZzMj .

[jxramos]

@segevfiner how would I have been subject to that, does VSCode or more specifically the vs-code python extension, do they ever prompt to install an update? I never uninstalled and reinstalled the application so I'm wondering if there's a log I could dig up that would indicate when I may have patched the system while I was running it. The numpy docstring formatting in my Before example is superb. It's a pity I can't figure out how to get back to it.

[segevfiner]

I think it auto updates in the background without asking and applies the update on restart/reload. There might be old versions available in the GitHub releases page that you can try to install to check.

בתאריך 23 במרץ 2018 10:50,‏ "jxramos" notifications@github.com כתב:

@segevfiner https://github.com/segevfiner how would I have been subject to that, does VSCode or more specifically the vs-code python extension, do they ever prompt to install an update? I never uninstalled and reinstalled the application so I'm wondering if there's a log I could dig up that would indicate when I may have patched the system while I was running it. The numpy docstring formatting in my Before example is superb. It's a pity I can't figure out how to get back to it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vscode-python/issues/38#issuecomment-375570769, or mute the thread https://github.com/notifications/unsubscribe-auth/AXlg_zbEyRO20EoR7gCM6jFwriTOq3F9ks5thKk3gaJpZM4QZzMj .

[brettcannon]

@jxramos VS Code automatically updates all extensions as @segevfiner says. They are also right that we have tried to clean up the output due to the fact VS Code wants Markdown but Python code typically uses reStructuredText.

If you want to try and find where things changed, you can look at https://github.com/Microsoft/vscode-python/releases and build the extension as appropriate.

[erichiller]

Adding to this issue that numpy docstring links (which as far as I know are reST) standard do not render.

[PhilMacKay]

I'm not sure whether it should be in a new issue, but using Jedi as IntelliSense engine instead of Microsoft Python Analysis Engine, I can get SciPy and NumPy docstrings to display nicely. Using the language server (by setting jediEnabled to false), I get almost no formatting for the same docstrings.

With Jedi:

With PyLS:

Can PyLS be configured to return formatted docstrings?

[edit: wrong screenshots!]

[brettcannon]

@PhilMacKay the formatting isn't in right now as we have not decided how to handle formatting overall (e.g. the only reason Jedi looks like that is it's assuming Markdown while a lot of Python docstrings use reStructuredText, so it's not an easy problem to solve).

[PhilMacKay]

Thanks for the fast reply, I understand the challenge. I'll be looking forward to the moment you solve this problem, as I like the recent progress on PyLS! You have probably looked at these already, but from what I read today, it seems the whole Python community is moving towards reStructuredText, including NumPy/SciPy. It would be a safe bet to add support for it. NumPy docstring format: https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard PEP about docstring format: https://www.python.org/dev/peps/pep-0287/

Thanks!

[yzhang-gh]

@brettcannon

The formatting isn't in right now as we have not decided how to handle formatting overall (e.g. the only reason Jedi looks like that is it's assuming Markdown while a lot of Python docstrings use reStructuredText, so it's not an easy problem to solve).

I totally understand. But can we at least do something to start the ball rolling? This issue is ranked 7th the most upvoted request in repository. So it is not that unimportant somehow.

Problem: Have not decided how to handle formatting (the scheme)

Premise: VSCode only accepts Markdown for docstring.

Do you wish to have an unified docstring scheme? or support different schemes (syntax) for different packages?

(IMHO, the second one is more practical.)

If so, which packages would we support?

• Python standard libs, of course (PEP 287, reStructuredText)
• Data science: Numpy, Scipy, Matplotlib, ... (...)
• Web?: Django, ... (...)

Proposal

• Make a default docstring "parser" (reStructuredText -> Markdown) I guess we already have it
• Allow specifical "parsers" for certain packages, so people can contribute to them. They may look like some messy patches. But having some patches is better than nothing. (e.g. the original naive parser)

Then, the core question is, can we know which package a docstring comes from? If so, could you expose a good API so other people can get involved in for the packages they need?

Many thanks.

[brettcannon]

@yzhang-gh it's not that simple as the formatting comes from the language server and that also affects Visual Studio.

And we are aware that this is a popular request and we are not purposefully ignoring it, but we also have popular issues 1-6 to deal with as well. 😉

[yzhang-gh]

Fair enough. Thank you for the information ❤️

[sleon-fmi]

Hi everyone!

Slightly related to this, I have been searching online for an extension (or support within VS Code itself) that can pick up docstring comments on module constants. e.g.:

USERS_TABLE: Optional[str] = os.environ.get('USERS_TABLE')
"""The name of the Users table
"""

It is my understanding that Sphinx will read these comments and include them in its output (see this SO answer from 2013 for context).

Is there support planned for this? If not, I hope you would consider it.

[brettcannon]

@sleon-fmi please make a separate feature request (although that isn't supported by Python itself so I don't know how likely we are to support it).

[danalf67]

I think it's very important to have a formatting more like in PyCharm, call signatures and docstring params are much more readable there. In the signature there are line breaks between each parameter, And in the docstring each ":param :" text are suppressed (the reST syntax). When you click the data type of a parameter, you are also redirected to the documentation for that class.

[bjtho08]

Is there any development with this issue? Has anyone been assigned to this?

[brettcannon]

@bjtho08 not that I'm aware of as the language server team is working on some fundamentals to try and get into a position for us to make it the default experience. You can probably follow Microsoft/python-language-server#613 for updates.

[bjtho08]

@brettcannon What if I use jedi rather than Microsoft Language Server? Does that change docstring rendering?

[brettcannon]

@bjtho08 yes it does

[bjtho08]

@brettcannon okay, so I would have to bring this up on their github rather than vscode-python?

[brettcannon]

@bjtho08 nope, it's still done by the extension, it's just a different code path and thus requires different changes than compared to the language server.

[adsr303]

I think it's very important to have a formatting more like in PyCharm, call signatures and docstring params are much more readable there. In the signature there are line breaks between each parameter, And in the docstring each ":param :" text are suppressed (the reST syntax). When you click the data type of a parameter, you are also redirected to the documentation for that class.

In PyCharm you can configure which docstring format you use in the project, including plain text. This affects both display and automatic docstring generation.

This makes me think an option to disable docstring formatting (except dedent like in help()) could be an acceptable stopgap measure.

[Ren4rd]

I am using some kind of docstring I adapted to render almost correctly with Microsoft Language Server... Almost because snake_case is somewhat not supported and break the rendering (which is reaaaaally bad :( )

cf : this issue I wrongly filled in vscode

[memeplex]

This makes me think an option to disable docstring formatting (except dedent like in help()) could be an acceptable stopgap measure.

I came here with exactly this in mind. Yet there is the problem that it's not possible to resize the documentation popup to fit the hard-wrapped text without upstream changes, isn't it @brettcannon ?

Other than that I believe that do-nothing is a reasonable solution since developers write their docstrings in order to be able to read them later in plain code. So it's not as cool as formatted output, but it avoids the many ugly ways the formatter gets broken now.

[Arcitec]

Well, since there are like 10 bajillizon gorillions different format for Python docstrings (ugh), there's no universal way to detect how to format them. So I think vscode-python is doing the right thing when it's doing some basic monospace display of the text in the tooltip! Otherwise, trying to auto-detect the string format would lead to all kinds of buggy rendering mistakes.

Also, as for the "automatically generate docstrings", I agree, it's a great feature, but there is already an extension for it: https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring

[memeplex]

So I think vscode-python is doing the right thing when it's doing some basic monospace display of the text in the tooltip

Problem is, it's not doing exactly that, except at parameter hints, for which the tooltip is too small for it to work (hard breaks appear anywhere). So we're in a sense in worst of both words here: the documentation popup is large and poorly formatted, the parameter hint popup is too small and unformatted (to make this even worse the font is unnecessarily big). Bigger popups, smaller fonts and no formatting would be a saner approach IMO.

[Arcitec]

@memeplex Ah yes true, it's definitely doing a bad job with line breaks of preformatted text.

[yajo]

there are like 10 bajillizon gorillions different format for Python docstrings (ugh), there's no universal way to detect how to format them

Wouldn't a config option serve? It can be stored per user or per project, so it seems reasonable:

// ${workspaceRoot}/.vscode/config.json
{
  ...
  "python.docstring.parser": "sphinx", // or others
  ...
}

Then wherever there's some code that parses those, it just should do this pseudo-code:

try {
  docstring = parse_dosctring(config["python.docstring.parser"], raw_docstring);
} catch (error) {
  docstring = raw_docstring;
}
show_tooltip(docstring);

[onacrame]

Agree this is incredibly annoying. Docstrings are just complete run-ons without new line breaks. Please fix.

[mcmx]

Any update on this?

[brettcannon]

@mcmx we don't give ETAs on feature work, so unfortunately no.

[RichardPflaum]

It would be amazing to have latex formulas in python docstrings rendered inline too. The extension goessner.mdmath has a solution for rendering latex, maybe this could be extended to inline docstring display?

[sleighsoft]

Would really appreciate to see improvements in rendering docstrings :)

[rben01]

Ideally, if/when implemented, the Intellisense window would understand the semantics of the docstring and not simply render a pretty-printed version of it. That is, the Intellisense window should be able to interpret docstrings and create a format-agnostic data structure containing the docstring's information, e.g., {"desc": "my function", "parameters": [{"name": "x", "type": "int", "optional": true, "default": 0, "desc": "my x param"}], "returns": {"type": "bool", "desc": "returned boolean"}}, and then turn this into a nicely-formatted view containing the same information. This way users don't need to see the markup text, and additionally can set flags that alter how the docstring is rendered.

[Evpok]

@rben01 Please don't advocate for a Python version of Javadoc

[rben01]

@Evpok Sphinx exists so there already is a python version of Javadoc. What I'm proposing though is that the VSCode Intellisense window not display markup in the window because who wants to see markup when reading documentation? Markup only exists for writers; readers should see the effect of the markup but not the markup itself.

[Evpok]

Of course formatting markdown should not be displayed, but what you are suggesting goes further. Intellisense targeted to Sphinx or similar is not a bad idea, but I would rather have proper markdown/rst parsing first

[memeplex]

It's important to support numpy convention which is very popular and currently broken in a bad way by blind markdown rendering of every docstring. The most evident problem is that parameter description renders as verbatim/code because of its indentation.

[brettcannon]

I am going to kindly ask that people not leave comments suggesting design ideas as there are 24 people receiving notifications on every comment left. I can lock the issue but that will freeze voting, so I would rather avoid doing that.

[Brodersen]

The current hover info is not very practical.

It would be really helpful to see just some (even the unformatted) docstring, until a better solution has been found. Or at least make it an option.

[MikhailArkhipov]

@Brodersen - the display is provided by the respective language server and not the core extension. If it is Pylance you can file bugs at https://github.com/microsoft/pylance-release, Microsoft LS: https://github.com/Microsoft/python-language-server.

AFAIK print comes from typeshed stubs that doesn't have doc strings.

Thanks.

[jakebailey]

For Pylance, this is: microsoft/pylance-release#49

[bradennapier]

The best I could personally come up with is pretty custom but it is actually pretty practical. By mapping back ticks to ' you could parse it as yaml. Not sure what the token is that makes that all red text, haven't tried to customize that yet.

[LucasRxTx]

Would love to see more support for parsing of doc strings. Would be interested in the plug in picking out basic configuration from sphinx, if sphinx is installed and configured https://www.sphinx-doc.org/en/master/usage/configuration.html. This would allow the vscode-python parser to know if it should expect google, numpy, sphix format, ect...

Having the hover feature would be awesome to be able to preview how it would render when/if the docs are built, and improve overall quality of doc strings for use in automatic documentation generation.

I see the issue is from 2017, but I have been wishing for this feature for far longer 😇

[jakebailey]

FWIW, in Pylance we are currently working on a big docstring update which should make numpy/google style docstrings look quite a bit better.

Calling out to python to ask for information is not something we really want to do, as we still need to be able to handle docstrings in every other code you might use too (dependencies), where looking at some local config would not be correct, since you yourself may be using sphinx format, but import numpy and hover its functions. (Also, the config is Python that is typically run with exec (!!) so may not actually be usable statically.)

Generating docstrings is another thing entirely than just the parsing and requires more care.

[jakebailey]

Since this was transferred, I'm going to close this; my update above was some 9 months ago and Pylance's docstring handling has been much improved. Any specific issue should be opened as a new issue.