auto-link functions, classes and members in docstring text

[rdzman]

Summary

As part of the goal of supplying in-source documentation that looks good and works well with both Sphinx and MATLAB's help and doc commands (see #140), it would be ideal if links to functions, classes, properties, methods, etc. could be auto-generated.

That is, for example, if the name of any item that is being autodocumented is found anywhere in the docstring it would be turned into the appropriate link. E.g. Writing This uses a myPkg.myClass object to ... would be automatically converted to This uses a :class:`myPkg.myClass` object to ... and rendered by Sphinx as a link.

Background

Currently, MATLAB's doc (or help) command automatically does two things:

1. Converts names to links in two contexts: (a) A line in the help text (docstring) that begins with See also. If there are multiple such lines, it only acts on the last one. (b) Property and methods names for MyClass listed under the "MyClass Properties:" and "MyClass Methods:" headings, repectively, in the class docstring.
2. Highlights (bold/colored) the name of the current class, function, etc, without making it a link.

With Sphinx any linking or highlighting must be done manually (e.g. using :class:`MyClass` or even ``my_variable``). This creates 2 problems when displayed by MATLAB's doc or help.

Problems

1. It breaks any auto-linking or highlighting mentioned above.
2. It adds clutter by including these Sphinx roles in the documentation displayed by MATLAB.

Current Alternatives

So currently, I have to choose between (1) non-linked, non-highlighted names in the Sphinx generated docs, or (2) no auto-linking and lots of ugly Sphinx-role-clutter in the MATLAB display.

I don't want to have to pick one or the other, especially in the See also lines, where I'm currently duplicating the line to get links in both contexts.

Questions

I'd love to hear your thoughts on the overall idea. And here are some specific questions regarding the possibility of this package including new auto-linking functionality sometime in the not-too-distant future:

1. Is there any possibility of at least creating auto-linking for See also lines?
2. What about for the properties and methods in the class docstring described in 1(b) above?
3. If those are possibilities, could we extend it to auto-linking everywhere in the docstring?
4. Would it help if we limited the auto-linking to fully qualified class and function names, excluding property and method names?

... or ...

5. Should I assume that links in Sphinx documentation will only ever be those created by using explicit roles in the source documentation? 😦

[rdzman]

Below are some of screen shots based on an updated minimal example I'm working with (classdef-min-ex2.zip). The ZIP file includes HTML generated by Sphinx (in build/html) and MATLAB's doc, via the undocumented help2html (in build/matlab).

MATLAB help

MATLAB doc

Sphinx

[joeced]

It's on my todo list as well, I thought of this for a long time. As soon as I get #171 done, it will be easier to make these kind of features.

[rdzman]

For the writing of my documentation, I'm currently leaning toward leaving out all explicit Sphinx roles, to avoid that clutter in the MATLAB doc and help output, with the hope that one day we can get this implemented. In the short-term my Sphinx docs won't have any special highlighting or linking of classes, functions, etc., but a simple re-building of the docs at a later date can hopefully add them. That way I won't have to go back and make changes to my documentation.

I'm also starting to look into adding this feature myself, beginning with just the See also lines. I just need to find how to look up the object from the name to decide what role to use, or whether to just use double back-ticks (in case it's not something we can link to ... e.g. a MATLAB built-in).

[rdzman]

Here's a first crack at auto-linking known classes and functions in See also lines. That is, any line in any docstring that begins with See also will wrap known names with a :class: or :func: role. For example, if MyClass, mypkg.MySubClass, and my_function are known entities, then the lines ...

See also MyClass, mypkg.MySubClass,
my_function, some_other_function.

... are converted to ...

See also :class:`MyClass`, :class:`mypkg.MySubClass`,
:func:`my_function`, some_other_function.

I do this by adding the following line ...

docstrings = self.auto_link(docstrings)

... just before line 204 of the add_content() method of MatlabDocumenter. https://github.com/sphinx-contrib/matlabdomain/blob/13dd4c0d5a5c79a7a2582bdbaca2162125538c65/sphinxcontrib/mat_documenters.py#L196-L205

And I define the new auto_link() method as follows.

def auto_link(self, docstrings):
    # autolink known names in See also
    see_also_re = re.compile("See also\s+(.+)")
    see_also_line = False
    for i in range(len(docstrings)):
        for j in range(len(docstrings[i])):
            line = docstrings[i][j]
            if line:                    # non-blank line
                if not see_also_line and see_also_re.search(line):
                    see_also_line = True    # line begins with "See also"
            elif see_also_line:         # blank line following see also section
                see_also_line = False       # end see also section

            if see_also_line:
                for n, o in entities_table.items():
                    if isinstance(o, MatClass):
                        role = "class"
                    elif isinstance(o, MatFunction):
                        role = "func"
                    else:
                        role = None
                    if role:
                        nn = n.replace("+", "")     # remove + from name
                        # escape . and add negative look-behind for `
                        pat = "(?<!`)" + nn.replace(".", "\.")
                        line = re.sub(pat, f":{role}:`{nn}`", line)
                docstrings[i][j] = line
    return docstrings

Note that this relies on the entities_table from #171.

Right now it only works on the fully qualified names of classes and functions.

I also tried doing the substitution on every line in docstrings, which I think we will eventually want to do. However, we will want to think this through carefully to avoid linking anything incorrectly (e.g. linking constructor name to class objects), or linking names in certain contexts, such as property and method section headings.

I think we will also want to generate auto links for properties and methods, but I wasn't sure where to find the corresponding objects as I don't see them in the top-level of entities_table.

I thought I'd wait until #171 was merged to create a PR for this, but I'd love to hear thoughts or feedback.

[joeced]

Nice idea. I had the same line of thought regarding

I also tried doing the substitution on every line in docstrings, which I think we will eventually want to do.

I think you observation is correct, that one has to be careful not the link to everything in there.

Regarding #171, it's progressing quite fine. I got base-class linking to work! Now I just need to merge or rebase with master and finally get it into master.

[rdzman]

Update - I've pushed my current work on this to the auto-link branch on my fork.

Current status:

• Added a matlab_auto_link config option that currently takes 2 options, "see_also" and "all".
• Added a ref_role() method to MatObject types to return the role to use for references to each type of object.
• Need to update the regex to match only whole words. Right now if you have myClass and myClassFoo, it will replace myClassFoo with :class:`myClass`Foo, which isn't quite what we're looking for.

I'll rebase on master and create a PR as soon as #171 is merged.

[joeced]

@rdzman I have had the version 0.20.0 out as release candidate for quiet a while. I'm looking into actually releasing it. Do you have any objections to just get out?

Then we could maybe close this issue?

[rdzman]

Ship it!

And thanks for including this. For my use case, this and other updates you've included in recent months have improved this tool tremendously!