formatting conventions for function documentation

[stevengj]

Right now, the @doc help suggests;

  @doc """
    # The Foo Function
    `foo(x)`: Foo the living hell out of `x`.
  """ ->
  function foo() ...

In method documentation, wouldn't it be much more satisfactory () for the help command to automatically prepend the signature for foo before the documentation output as needed?

In generic-function documentation (e.g. documentation for sum in all its forms), obviously this is not possible (since there will in general be multiple possible signatures). However, we should establish some convention for what the documentation should look like.

[johnmyleswhite]

I've wondered if we'd need automatic stubs for documentation. One of the things I most like about R's documentation system is that individual-level documentation of every single parameter of a method. I'd really like to see that kind of detail in most documentation for Julia as it matures.

[ViralBShah]

Yes, having the signature automatically is certainly nice. Also, some structure and style that is common across Base and eventually across packages is also good to target for. Should we try to pick a few functions and establish what documentation should like ideally for those, and then bake that in?

[MikeInnes]

Automatically displaying the signature would be nice in some cases, but not all; e.g. if you define your user-level function as f(args...) and call a lower level routine, or Python, or whatever, the actual method signature isn't meaningful. Same for abbreviated names etc.

One thing I'd definitely like to see is some form of one-line summary convention. We can do all sorts of nice things with that, like intellisense-style autocomplete in IPython/Juno and even live, smart code hints, which I have a very rough prototype of:

That said, I'm also leaning towards letting standards evolve naturally rather than enforcing something early on. The result is likely to be better and using markdown means it's pretty trivial to pull out structured data even if the conventions are fairly loose.

[StefanKarpinski]

One convention I've thought about from time to time is to use long descriptive argument names. We do have a tendency to abbriviate especially on some additional method definitions, however, which is convenient; maybe we could have some system where foobarbaz could be abbreviated to fbb – i.e. by looking for long names on the first method defintion (or in the docstring itself?) and associating abbreviated argument names with the long names that they are subsequences of.

[hayd]

Perhaps see numpy for inspiration https://github.com/numpy/numpy/blob/master/doc/example.py https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt.

Numpy basically uses markdown but doesn't do anything special to extract sections:

@doc """
Overview of `foo` function.

Parameters
----------
`x`
    thing to be fooed.
""" ->
function foo() ...

perhaps this could be lowered/expanded to something like (although these probably need to be ordered):

@doc MetaDoc[:docstring => md"Overview of `foo` function",
             :parameters => [:x => md"thing to be fooed."]] ->
function foo() ...

In the syntax of https://github.com/JuliaLang/julia/pull/8588#issuecomment-60244983.

I think having this making actual metadata rather than just a convention will be useful... this way one could, say, extract all the references in a module, or highlight the doc of the current parameter as you type, or toggle through just code examples.

[ViralBShah]

These are the sections @milktrader suggested. These seem like a great start.

Description 
...
Usage
...
Arguments
...
Details
...
References
...
See Also
...

[johnmyleswhite]

In addition to Arguments, we need Returns as well.

[stevengj]

Compare to the documentation we have now, where a typical function has a one-paragraph description.

Would it really be readable if every single one of these descriptions were expanded into 7 subsections complete with headings? I'm skeptical.

[johnmyleswhite]

Coming from R, I find the current documentation a little too terse. For simple functions, I agree that leaving out many of those subsections seems reasonable. But for functions with even the slightest bit of subtlety (e.g. mad in StatsBase), the thoroughness of R's documentation strikes me as the right endgoal. Here's the docs for mad in R, which I think have the right level of information:

mad                   package:stats                    R Documentation

Median Absolute Deviation

Description:

     Compute the median absolute deviation, i.e., the (lo-/hi-) median
     of the absolute deviations from the median, and (by default)
     adjust by a factor for asymptotically normal consistency.

Usage:

     mad(x, center = median(x), constant = 1.4826, na.rm = FALSE,
         low = FALSE, high = FALSE)

Arguments:

       x: a numeric vector.

  center: Optionally, the centre: defaults to the median.

constant: scale factor.

   na.rm: if ‘TRUE’ then ‘NA’ values are stripped from ‘x’ before
          computation takes place.

     low: if ‘TRUE’, compute the ‘lo-median’, i.e., for even sample
          size, do not average the two middle values, but take the
          smaller one.

    high: if ‘TRUE’, compute the ‘hi-median’, i.e., take the larger of
          the two middle values for even sample size.

Details:

     The actual value calculated is ‘constant * cMedian(abs(x -
     center))’ with the default value of ‘center’ being ‘median(x)’,
     and ‘cMedian’ being the usual, the ‘low’ or ‘high’ median, see the
     arguments description for ‘low’ and ‘high’ above.

     The default ‘constant = 1.4826’ (approximately 1/ Phi^(-1)(3/4) =
     ‘1/qnorm(3/4)’) ensures consistency, i.e.,

                         E[mad(X_1,...,X_n)] = sigma                    

     for X_i distributed as N(mu, sigma^2) and large n.

     If ‘na.rm’ is ‘TRUE’ then ‘NA’ values are stripped from ‘x’ before
     computation takes place.  If this is not done then an ‘NA’ value
     in ‘x’ will cause ‘mad’ to return ‘NA’.

See Also:

     ‘IQR’ which is simpler but less robust, ‘median’, ‘var’.

Examples:

     mad(c(1:9))
     print(mad(c(1:9),     constant = 1)) ==
           mad(c(1:8, 100), constant = 1)       # = 2 ; TRUE
     x <- c(1,2,3,5,7,8)
     sort(abs(x - median(x)))
     c(mad(x, constant = 1),
       mad(x, constant = 1, low = TRUE),
       mad(x, constant = 1, high = TRUE))

[ViralBShah]

@stevengj If we have section templates, we can choose different levels of verbosity for the command line help (just show description and usage), and the manual which has all sections. Matlab also does more than what we do currently, which helps when one is new to the system.

[stevengj]

Fair enough.

[alanedelman]

I've always wished one could click somewhere and find out about numerical notes ....when available. Some things would be links to issues about accuracy, even numerical comparisons to other systems, links to Kahan wisdom, etc. In other words, the help becomes more than how to use Julia, it becomes a whole numerical course.

[ViralBShah]

I have always wished that too. With the Julia community, we actually have the skill to pull this off, and hopefully even create something that is more comprehensive and better than what any other system of this sort has done before.

[stevengj]

(Ideally, just click to edit the documentation and then submit a pull request.)

[jlapeyre]

I implemented documentation for the Maxima CA (as third party code), Much of this is already being done for Julia, but I wanted to get some ideas out. I did it all myself; it would great to see Julia collaborators do something similar and more ambitious, and do it right.

https://github.com/jlapeyre/mext/tree/master/packages/defmfun1

Examples of using the system are in several packages in the sibling directories.

I (almost) never write things like "foo(x) does...", or " takes a string x as input, and returns ...", or "Examples: ". The function name, arguments and types and special restrictions, examples, and other features are all meta data, when possible extracted automatically or with minimum specification. It is used to generate uniform, consistent documentation. The same meta data is used for argument checking and for printing (or suppressing, statically or dynamically, globally or per function) uniform, consistent error and warning messages.

Here is (outdated) html output of the documentation for several functions that I wrote with this system.

http://www.johnlapeyre.com/thirdparty.html

Many Maxima user-level functions are implemented as (Common) lisp functions. It is standard practice to handle argument-checking, argument pre-processing, error messages, documentation, etc. in an ad hoc way. I wrote a macro, 'defmfun1' to wrap 'defun'. The macro arguments ("directives") affect the behavior of the function as well as the generated meta-data. All the special behvavior can be omitted, or easily turned off either in the definition, or at run time.

It is typical for Maxima functions to check arguments, kind of like Mathematica. This is done in an erratic and ad hoc way. I liked the more-or-less consistent Mathematica error (or warning) messages and getting them easily and consistently was attractive. I also like the uniform and consistent Mathematica documentation. I used the Mathematica messages as a model.

Here are up-to-date examples:

---

(defmfun-ae ($random_cycle :doc) ((n :pos-int :thread)) [ threading loops are generated automatically ]
  (rand-perm-sym-body t)) [ This macro writes one of two function bodies depending on second arg ]

     [ The markup has to be more or less compatible with texinfo to integrate with Maxima.
       You can add several call descriptions which are listed one after the other. These are
      like the short descriptions at the top of an Mma documentation page. ]

(add-call-desc '("random_cycle" ("n") 
   ("Returns a random cycle of length " :argdot "n" " The return value is a list
     of the integers from " :math "1" " through " :argcomma "n" " representing an
     element of the symmetric group " :math "S_n" " that is a cycle.")))

(max-doc:implementation "random_cycle" "This function uses Sattolo's algorithm.")

 [ Each of these "see alsos" all others ]
(max-doc:see-also-group '( "random_cycle" "random_permutation_sym" "ae_random_permutation"
                           "signature_permutation" "perm_to_cycles" "cycles_to_perm"))

---

Interactive session. Print doc and try examples:

(%i14) ? random_cycle

  -- Function: random_cycle: random_cycle(<n>)
     Section: Functions and Variables for Combinatorics
     mext package: discrete_aex; filename: combinatorics.lisp

Calling:
   random_cycle(<n>)
        Returns a random cycle of length <n>. The return value is a 
        list of the integers from 1 through <n>, representing an 
        element of the symmetric group S_n that is a cycle.

Arguments:
   `random_cycle' requires one argument <n>, which must be a positive integer.
    The first argument threads (distributes) over lists.

Options:  random_cycle takes options with default values:
          adj->true, ot->ml.
Attributes: random_cycle has attributes: [match_form]

See also:  random_permutation_sym, ae_random_permutation, signature_permutation, perm_to_cycles, and cycles_to_perm.

Implementation:
       This function uses Sattolo's algorithm.

(%o14)                               true
(%i15) random_cycle([3,4]);
(%o15)                     [[3, 1, 2], [3, 1, 4, 2]]
(%i16) random_cycle([3,"dog"]);
  Warning: random_cycle: Argument 'dog' at position 1 is not a positive integer in random_cycle("dog").

(%o16)                  [[3, 1, 2], random_cycle(dog)]
(%i17) random_cycle(1,2,3);
  Warning: random_cycle: random_cycle called with three arguments; one argument is expected.

(%o17)                       random_cycle(1, 2, 3)

---

Here is code that generates examples in the documentation. The data could also be used for interactive examples (a la Mathematica):

(examples::add-example "lrange"
        '(:code  ("lrange(6)" "lrange(2,6)" "lrange(2,6,2)"
                  "lrange(6,1,-1)" "lrange(6,1,-2)" 
                  "lrange(6,ot->ar)"))
        '(:pretext "The type of the first element and increment determine the type of the elements."
          :code ("lrange(1.0,6)" "lrange(1.0b0,6)" "lrange(1/2,6)"
                  "lrange(6.0,1,-1)"))
        '(:pretext "Symbols can be used for limits or increments."
                   :vars "[x,a]"
                   :code ("lrange(x,x+4)" "lrange(x,x+4*a,a)")))

The plain text format is:

Examples:
   (%i1) lrange(6);
   (%o1) [1,2,3,4,5,6]
   (%i1) lrange(2,6);
   (%o1) [2,3,4,5,6]
   (%i2) lrange(2,6,2);
   (%o2) [2,4,6]
   (%i3) lrange(6,1,-1);
   (%o3) [6,5,4,3,2,1]
   (%i4) lrange(6,1,-2);
   (%o4) [6,4,2]
   (%i5) lrange(6,ot->ar);
   (%o5) <<[1,2,3,4,5,6]>>

   The type of the first element and increment determine the type of 
   the elements.

   (%i1) lrange(1.0,6);
   (%o1) [1.0,2.0,3.0,4.0,5.0,6.0]
   (%i1) lrange(1.0b0,6);
   (%o1) [1.0b0,2.0b0,3.0b0,4.0b0,5.0b0,6.0b0]
   (%i2) lrange(1/2,6);
   (%o2) [1/2,3/2,5/2,7/2,9/2,11/2]
   (%i3) lrange(6.0,1,-1);
   (%o3) [6.0,5.0,4.0,3.0,2.0,1.0]

   Symbols can be used for limits or increments.

   (%i1) lrange(x,x+4);
   (%o1) [x,x+1,x+2,x+3,x+4]
   (%i1) lrange(x,x+4*a,a);
   (%o1) [x,x+a,x+2*a,x+3*a,x+4*a]

---

There is a table of argument specifications used for inserting code, for argument checking, printing messages, and documentation:

https://github.com/jlapeyre/mext/blob/master/packages/defmfun1/arg-spec-definitions.lisp

This kind of table would probably be used differently with Julia, eg called explicitly, since the multiple dispatch model prevents inserting the code at a single point. But, this meta-data can still be used both for errors/warnings as well as documentation.

I implemented uses for some other meta-data that may be relevant for Julia.

• authors, copyright; manual section; reference for algorithm, implementation notes, "see also", etc. Of course not all of this information is displayed always for everyone. Maybe someone never wants to see copyrights or implementation notes.
• index into external data base (OEIS, etc.)
• conditional content for plain text, latex math, etc.
• having a simplified version of metadata entry for non-expert users. Just simple strings, for instance. But, it should be compatible with and well-integrated with the more sophisticated entry methods. I wrote something for use in the Maxima language.

I tried to find an simple way to include the data in or near the function definition. I only used s expressions, and was only partially successful in making readable entries, although I think with effort this could be improved. Then, using the data (or not!) is a separate issue.

Some issues/differences with Julia

Building the doc structures significantly slows compilation. I compiled the lisp code, so the meta data is only collected and processed once. Does not work directly with jit. Maybe have a flag on installation that processes the meta data and writes to a per module database. For normal use the doc macros are ignored. But, this is a much more complicated installation than just copying jl files.

multiple dispatch: how to handle documentation coding and use when methods are added for a function? Where does the documentation belong if the methods really are doing very different things. And you can't extract meta data for docs as easily from a parameter list if you use multiple dispatch rather than argument conversion within the function. Using logic on all existing methods sounds complicated. Mathematica dispatches on patterns, how do they do it ?

[artkuo]

Any chance of adding something like Mathematica's REPL-aware function templates within the doc structure? Their notebook front end parses the "usage" docstring, and provides a template when you use the "make template" shortcut (cmd-shift-k on OSX), e.g. Then if you click paste, the template is inserted and you can tab between and fill in the fields: The Mma implementation is pretty good and automatically seeks the template, and gracefully degrades when no template is provided. It can also handle multiple templates, useful for multiple dispatch. There's also option templating, which I've never tried. All of this is available to user-defined functions, so it need not be baked into an IDE. I believe most of this would already be possible from the ideas I see above.

Another thought is to have the docs include more explicit argument definitions or have clearer parsing rules, rather than MMa's ad hoc method. This could potentially allow for things like tooltips pop up when you hover over an argument x to show expanded documentation like "a numeric vector". One annoyance with Mma is that once you are inside the template, you can't hover or go back to the usage string to be reminded about what x was. With a slightly stricter documentation format, it's possible to do more at the REPL level.

[jiahao]

In #9447 @MichaelHatherly @one-more-minute @hayd very kindly humored me as I ranted about the features I wanted in documentation.

I have just reformatted in IterativeSolvers: rlinalg.jl and rsvd.jl the documentation of rsvd, rnorm, reigmax, reigmin, rcond, IterativeSolvers.rrange, IterativeSolvers.rrange_si, IterativeSolvers.rrange_adaptive, and other functions to comply with the new @doc doc "" -> format.

It's really nice to see help for functions I wrote!

I hope my usage helps clarify why I would like inline cross-references, equation rendering and bibliographies.

[hayd]

I was having a play trying to put together a MetaDoc type a'la @stevengj in https://github.com/JuliaLang/julia/issues/8514#issuecomment-57331690 and https://github.com/JuliaLang/julia/pull/8588#issuecomment-60244983, it doesn't seem to play well with @doc I think this is since you can't override mdify (but no doubt I'm missing something)....

Anyways a gist is here: https://gist.github.com/hayd/bdddb51dc7df89b9a24e

Thinking about it may be that you could just extract sections from the text i.e. lower @jiahao's docstrings into MetaDoc... (kind of docopt-style).

There's some discussion about Returns section (and how you could use this) in PR to prepend signature info to the help message: https://github.com/JuliaLang/julia/pull/9838.

[hayd]

I hadn't realised that Docile has a meta object, but that didn't make it to base.

@MichaelHatherly IMO Docile.meta ought to be in base, having two levels of documentation (one in Base and one in Docile) seems strange. getting it in base would help flesh out what meta is and get it working (e.g. it doesn't work with help for me in 0.4 - following the examples in the readme :( ). Pretty please can we have meta!

[MichaelHatherly]

@hayd, I think I might have added meta some time after the @one-more-minute's initial docs pr, so that's probably why it got missed.

From the looks of your gist that's basically all there. Not sure about the rendering trouble you're having; @one-more-minute is the best person to ask about that. Some differences I can see: naming (I can easily change names in Docile if we settle on MetaDoc rather than meta), the meta entries are stored as a Dict{Symbol, Any} rather than Vector{Pair} in Docile. I think the => syntax is 0.4 only if I'm not mistaken, in which case I can't really support it in Docile.

[hayd]

@MichaelHatherly You may be able to use the compat module to support 0.4 syntax on 0.3.... ?

TBH I'm not sure what syntax is best here going forward anyway, perhaps others have some ideas, but :100: to getting meta in. :)

[MikeInnes]

I want to point out that extensions, like meta, can both live in external packages and be first-class parts of the doc system. In fact, this was a key design goal; I don't want to mandate how documentation is done so much as allow people to do whatever they might need. There are other advantages to keeping the non-core functionality in packages, too, like the fact that you can release updates whenever.

So once 0.4 is out there's really no reason Docile.jl can't be compatible with it. I'm sort of hoping it will live on as a way to have all of the great "advanced-mode" doc features that it currently supports, including things like metadata.

[prcastro]

I've been writing some docs with this system and, alongside metadata, I would find very useful to have a way to insert the function's signature into the docs automatically. Actually, the same for constants (insert their types into docs), types (insert their supertypes), macros and generated functions, etc. Maybe something like:

@doc """
$signature$

Foo the living hell out of an Int
""" -> function foo(x::Int)

So that:

julia> ?foo(10)
foo(Int)

Foo the living hell out of an Int

Also, I really believe metadata for docs should be on Base. This would incentive developers to write richer documentation from start, while guaranteeing good documentation on how to do this, and a single stardard way to do it.

[hayd]

There;s been some major changes to Docile of late, which parses

"docstring with *markdown*"
function f(); end

even if there's no using Docile in the package/file. It can even do this on base! see https://github.com/MichaelHatherly/Docile.jl/issues/56#issuecomment-108759557

I think there is an issue about signature (and I had a PR rejected), IMO it should always be shown. `$ wouldn't work as it's already used for latex, but there may be some other options for similar things...

I think the metadata will work itself out, now that Docile doesn't need using... most packages use ### Arguments or #### Returns, so these could be silently extracted into metadata (if there was some benefit to do that). When base moves to markdown this convention should be followed.

[ScottPJones]

Well, why not just use the javadoc/doxygen/etc. tags (they are pretty standard), instead of coming up with something julia specific?

##\brief Brief description here
#
# detailed description
# \param[in]    str::AbstractString  String to be munged
# \param[out]  buf::Vector{UInt8}  Output buffer
#
# \returns        Bool   if string was valid or not
# \throws         ArgumentError  (if str is empty)

Doxygen supports Markdown, and the comments can be easily filtered to remove the \ and colorize the param, etc. tags, and remove totally the \brief tag...

I think this would be better than everybody coming up with their own incompatible style, which is what is going on now...

[prcastro]

Parsing markdown to extract metadata sounds awesome. We could parse anything before an (sub)title as a overview end xtract other section based on title names. For parameters, I believe we should have a standard way to do it, maybe a markdown list of parameters and their types/descriptions:

### Arguments
* x: A number to be fooed
* `y`: Simply *anything*

Would get us something like:

:arguments => Dict(:x => "A number to be fooed", y: => "Simply *anything*" ])

Down the road we could use the upcoming Ordered Dicts to provide the order of these parameters while still using a Dict.

An IDE could, then, use this info (alongside function's signature) to provide tooltips for each parameter as we type them (like their types and descriptions).

[ScottPJones]

Yes, but just using the \param tag, with the optional [in], [out], [in,out] attributes, makes it really easy to pull out all of that and create the metadata, for \returns and \throws as well.

[ScottPJones]

Instead of an OrderedDict, what about the NamedTuples that I've heard about?

[prcastro]

This may work, but I'd prefer not to add new syntax just to obtain metadata. More syntax to document, test and remember.

[MikeInnes]

@prcastro Yes, something like this would work well. The more I think about it the more I dislike the idea of specialised syntax like doxygen/javadoc.

I mean, I get it – we're programmers, and writing things that computers can parse and manipulate Just Feels Right. But doc strings have to be designed for humans first – and humans, particularly ones new to programming, are a lot more comfortable with paragraphs of text than noisy syntax trees and \param[in]s.

On top of that, we already have two syntaxes – Julia and Markdown – and we should be able to use them to specify anything we want (in the limiting case, by splicing a metadata dict into the docstring, for example). As long as structured data is a nice-to-have rather than a make-or-break, there's no way to justify adding a third syntax for people to learn on top of that.

[prcastro]

Exactly, things become even more difficult to the enduser when you pile up more syntax. Afterall, people searching for that kind of power could find it on packages like Docile. Base should have just a simple metadata system, that is as natural and as unobtrusive as possible to newcomers.

Would the overhead of parsing markdown to obtain metadata be a problem?

[MikeInnes]

I don't think so – we already parse markdown to display it so it's just a case of looking over the structure for metadata lists etc. Should be pretty easy, really.

I should clarify also that Base's default doesn't prevent anyone using doxygen or whatever if they really want. You can just use your own string macro:

dox"..."
foo ...

[ScottPJones]

@one-more-minute Ha! I already have a dox"..." macro! Might I say, "great minds think alike"? :grinning:

@prcastro You'd still have to come up with some new syntax to say if something is a parameter, a return value, or what errors are thrown... my point is simply, there is already a syntax that is pretty much a de facto standard, why not use that? I think a lot of people would probably be at least familiar with it.

I guess, if you actually do make sure that everything is correctly parsed, and you can produce the metadata for: brief description, full description, parameters, return value(s), and errors thrown, then to support doxygen (or any other documentation tool), you could just output the metadata converted to the doxygen rules... (along with all the other information it needs, like what line the method is on, the method signature, what functions it calls, etc). I'd drop my push for doxygen specifically, if somebody does get a standard for at least those "tags" in place, and is able to show a working parser that collects all of the metadata. Finally, what were you thinking for type or enum fields? That's handled by doxygen (and for me, is the second best benefit of using the tool... you simply put ##< instead of #, or """<...""". I suppose dox"..." would be enough to indicate it needs to be passed to doxygen, as long as it can pick up the context (i.e. that it is in a type, immutable, or enumeration declaration, the name of the field, the name of the type, and the file name and line number)

[StefanKarpinski]

For what it's worth, I'm 100% with @one-more-minute on this – adding doxygen syntax to the mix is causes more problems than it solves – and we want to support people before machines. The issue with generic documentation systems like doxygen is that they never quite do what you need, so you end up in this close-but-no-cigar situation until you throw in the towel and build/design something custom that's really tailored to your specific needs. Julia is unusual enough as a language that the chances of something "standard" like doxygen working really well for it seems negligible.

[ScottPJones]

You maybe missed my comment elsewhere, I am also now 100% behind @one_more_minute's excellent idea of parsing the meta data out of the doc strings, from which I could generate a file in a form doxygen could understand, to get all the x-references out of the code, even x-referencing with the C source files in julia/src

[StefanKarpinski]

I wasn't addressing you specifically.

[hayd]

Extracting metadata from markdown has the benefit of a graceful failure. Also checkout https://github.com/MichaelHatherly/Lexicon.jl ...perhaps we can use that to build the docs in the future (without migrating it to base!!)

[JeffreySarnoff]

@alanedelman wrote "I've always wished one could click somewhere and find out about numerical notes"

Last night I put an http link inside of @doc inline function documentation. It half worked. I was not able to use the usual markdown way Read and Click This. Writing the http address inside of the square brackets worked as a link, when I right clicked on it, a small menu popped up, "connect to the site" was a choice and it did that.

[hayd]

@JeffreySarnoff they're called autolinks. I did a PR for that a while ago. There's a way to go for full commonmark compat.

[JeffreySarnoff]

@hayd clearly appropriate

There is a watershed of benefit to Julia when, as one designs, experiments, codes, and dreams there are co-available threads of insight to push and enagement to pull. Soon there are others pulling and pushing through shared threads. This self-magnifies, setting a place for Julia in many, many places.