Does there exist a way to extract documentation?

[Kelet]

Hello,

I apologize if this isn't the right place for this, but this issue tracker seems to be more or less a mailing list at this point.

Anyway, I'm working on a small platformer 'engine' in LÖVE. It is my first foray into Lua, and I enjoy having the control. However, once I'm done with that, I'm going to be working on a game module. Since I'm very used to Ruby and other quite expressive languages, I've been considering Moonscript. It looks excellent.

My largest concern at this point is extracting documentation. My engine has pretty verbose commenting, and is nearly fully documented using LDoc to generate nice HTML pages. Moonscript does not seem to have an adaptation of LDoc (or any other system), and as I've read, comments are not translated to the generated Lua source, so that is also out of the picture. While my game having extractable documentation isn't nearly as important as my engine, it would still be preferable.

I might be missing something, or perhaps there is just no convenient way to do this yet. Was just hoping to get some input from Moonscript developers/users.

[leafo]

There's nothing like this yet, but I do think it's a good idea.

There are two options: A MoonScript specific version that extracts comments. I could see this working well for documenting classes. Or preserving the comments when compiling and then running LDoc on the output. Would LDoc be able to handle MoonScript's compiled output or are there strict rules about how the code is structured?

[Kelet]

Would LDoc be able to handle MoonScript's compiled output or are there strict rules about how the code is structured?

LDoc would be able to handle MoonScript's compiled output; however, I don't believe just copying the comments is going to be conducive to a good experience.

One of the reasons I like LDoc, and pretty much any modern documentation extraction systems, is that I can usually just place comments where I'd normally place them, and the system will pick up on that and intelligently figure out what I mean. Sometimes I need to be somewhat explicit by providing an extra tag or 2, but usually it's pretty good.

Unfortunately, some parts of MoonScript would need a rather specific and explicit comment structure to pass through correctly to the compiled output. From my limited experiences, classes would be such an example. In Lua with LDoc, all functions that look like ClassName:functionName() under a type ClassName tag will be put in the ClassName class in the resulting documentation (until EOF or another section). With your system, you would need something like within ClassName above each function, due to the fact that the functions are held in a table. And also it might be worthful to use the function directive to manually name it with a class prefix. It may not be obvious to someone who is used to LDoc that he needs to do this, and it's pretty verbose. (This is all untested conjecture however).

I would say that a MoonScript-specific version, or preserving the comments with perhaps some intelligence on where to put them in some cases is a good idea. I'll have to look into it more closely soon, maybe manually copy over the comments and see how it works out.

[stevedonovan]

I'm looking at a new LDoc release at the moment, and it would not be difficult to teach LDoc enough Moonscript to make it useful. It could infer function names from the usual pattern name = [(...)] (-|=)> ... and cope with class. I'll try for a first version in the next day or so.

[stevedonovan]

@Kelet it would be very useful if you could send some representative examples of Moonscript + documentation!

At least LDoc was designed with multiple parsing strategies in mind (currently Lua and C) so there's a framework for adding new languages.

[Kelet]

@stevedonovan I very much appreciate your efforts in attempting to add a new parsing strategy to LDoc for MoonScript. Unfortunately, I don't really have any codebases with MoonScript (LDoc is kind of the rocking point for me using it or not). I'll try to whip up a few decent examples for you later in the week, unfortunately quite busy right now.

[stevedonovan]

No problem! The idea is to collect together some representative Moonscript styles and train LDoc to handle them.

For instance, this is a very cool way to define modules:

-------
-- simple moon module
-- @module simple

with {}

    --- nice greeting
    .answer = -> .quote 'hello'

    --- quote the string `s`
    .quote = (x) -> '"'..x..'"'

    --- star! star!
    .star = => '*'..@

And just dumping through latest ldoc gives this (note how the fat arrow is handled)

$ldoc --dump simple.moon
----
module: simple  simple moon module

function        answer()
nice greeting

function        quote(x)
quote the string `s`
parameters:
        x

function        star(self)
star! star!
parameters:
        self

Here's a more realistic version:

----
-- A Moon module exporting a List class
-- @module list
import insert,concat,remove from table

--- A list class that wraps a table.
class List

    --- constructor passed a table `t`, which can be `nil`.
    new: (t) =>
        @ls = t or {}

    --- append to list.
    add: (item) =>
        insert @ls,item

    --- insert `item` at `idx`
    insert: (idx,item) =>
        insert @ls,idx,item

    --- remove item at `idx`
    remove: (idx) => remove @ls,idx

    --- length of list
    len: => #@ls

    --- string representation
    __tostring: => '['..(concat @ls,',')..']'

    --- return idx of first occurence of `item`
    find: (item) =>
        for i = 1,#@ls
            if @ls[i] == item then return i

    --- remove item by value
    remove_value: (item) =>
        idx = self\find item
        self\remove idx if idx

    --- remove a list of items
    remove_values: (items) =>
        for item in *items do self\remove_value item

    --- create a sublist of items indexed by a table `indexes`
    index_by: (indexes) =>
        List [@ls[idx] for idx in *indexes]

    --- make a copy of this list
    copy: => List [v for v in *@ls]

    --- append items from the table or list `list`
    extend: (list) =>
        other = if list.__class == List then list.ls else list
        for v in *other do self\add v
        self

    --- concatenate two lists, giving a new list
    __concat: (l1,l2) -> l1\copy!\extend l2

    --- an iterator over all items
    iter: =>
        i,t,n = 0,@ls,#@ls
        ->
            i += 1
            if i <= n then t[i]

return List

And the result of ldoc -S -f markdown list.moon is here (That's -S for simple and markdown for doing the backticks nicely.)

It's not perfect - I would have prefered to output List\extend(list) instead of List.extend(self,list) but that's the fat arrow case kicking in. (It's hard to find out whether I'm inside a class when doing the parse phase.)

@leafo can you suggest a nicer style & template? Current style is still very much LuaDoc, and it's a new century now ;)

[exists-forall]

This doesn't have to be done at the language or third-party parser level. If you don't already have too much documentation that you'd have to change, you can adopt a system where you generate docs as you initialize your library by setting up a function that records documentation. Here's the simplest system:

do
  docs = {}
  export doc = (str, obj)->
    docs[obj] = str
    obj
  export get_doc = (obj)-> docs[obj]

thanks to Moonscript's braceless function calls, integrating this into your normal workflow is trivial. All you have to do is take what you normally write

foo = (...)->
  -- func body

Bar = class
  -- class body
  method: =>
    -- whatever

and add documentation where it's relavant

foo = doc "transmogrifies every odd argument, with its successor acting as a scaling factor", ->
   -- func body

Bar = doc "Represents a widget that has not yet been transmogrified", class
  -- class body
  methods: doc "prepares widget for transmogrification", =>
    -- whatever

Moonscript's syntax makes this very natural. You can use a number of different forms

foo = doc "documentation", ->
  -- body

foo = doc "documentation",
->
  -- body

foo = doc [[
  documentation 1
  documentation 2
]], ->
  -- body

foo = doc [[documentation 1
            documentation 2]],
->
  -- body

foo = doc [[
  documentation 1
  documentation 2]], ->
  -- body

The last one, I think, is preferable, since it adds exactly one line of source for every line of documentation.

This system really shines, however, when you want to extract the documentation. No fumbling with parsers or command line tools - just load up your library and call get_doc on everything you want the documentation of, all in pure moonscript! Here's a script that exports in markdown:

lib = require io.read!

for k, v in pairs lib
  print "# `#{k}`"
  print get_doc(v) or tostring v
  if 'table' == type(v) and v.__base
    for v_k, v_v in pairs v.__base or v
      print "## `#{k}#{if 'function' == type v_v then '\\' else '.'}#{v_k}`"
      print get_doc(v)
      print!
  print!

producing an output like

# `foo`
transmogrifies every odd argument, with its successor as a scaling factor

# `Bar`
Represents a widget that has not yet been transmogrified

## `Bar\method`
prepares widget for transmogrification

[stevedonovan]

Yeah, we were just now talking about how useful it is to have readily-available documentation in a REPL.

This approach reminds me of David Manura's similar proposal except of course Moonscript as always involves less typing.

However, there are some issues:

• you really do want to capture the whole signature of the function. Don't necessarily need to go mad with @param etc.
• every function carries documentation as code, using up heap. Since most uses of a library are non-interactive, this feels wasteful.
• comments are highlighted nicely - some editors can even be taught to recognize doc string patterns as special.

Now, in that discussion on lua-l, I agreed that ldoc is too heavyweight just for interactive purposes, but the LuaDoc style is really straightforward and easy to extract on an ad-hoc basis. It's just a comment starting with more than two hyphens before a function, class or method.

[exists-forall]

Those are all good points. The heap usage could be alleviated by having a docs_enabled flag, and making doc just ignore docstrings when it's false. The highlighting could be done to strings just as easily as comments, although there is the issue of unwanted highlighting in normal strings. Maybe it will only highlight in long-strings?

As for the function signature, the only I can think of is some serious voodoo with the debug, library. I think you can determine the arguments to a function from its bytecode, but that might be incompatible across different versions and implementations of lua.

On Tuesday, August 13, 2013, Steve J Donovan wrote:

Yeah, we were just now talking about how useful it is to have readily-available documentation in a REPL.

This approach reminds me of David Manura'shttp://lua-users.org/wiki/DecoratorsAndDocstringssimilar proposal except of course Moonscript as always involves less typing.

However, there are some issues:

• you really do want to capture the whole signature of the function. Don't necessarily need to go mad with @param https://github.com/parametc.
• every function carries documentation as code, using up heap. Since most uses of a library are non-interactive, this feels wasteful.
• comments are highlighted nicely - some editors can even be taught to recognize doc string patterns as special.

Now, in that discussion on lua-l, I agreed that ldoc is too heavyweight just for interactive purposes, but the LuaDoc style is really straightforward and easy to extract on an ad-hoc basis. It's just a comment starting with more than two hyphens before a function, class or method.

— Reply to this email directly or view it on GitHubhttps://github.com/leafo/moonscript/issues/105#issuecomment-22550433 .

[Kelet]

@stevedonovan I was able to check out a new revision of LDoc today, and it worked well with the small amount of MoonScript I have written. I should be adding more MoonScript code to my game soon, so I'll definitely follow up with some suggestions or whatever comes up when that happens.

@SelectricSimian I like that idea a lot. I may use it for some future projects. My current project is part Lua and part MoonScript though, so it's nice to have unified documentation for both types of source code with LDoc.

As for documentation in a REPL, Ruby and irb/pry seem to do that for installed gems that provide RDoc (which is similar to LDoc). Not completely sure how it's done, but I've used it many times before.

[TangentFoxy]

Has there been any more work on something like this or ideas or anything?

[stevedonovan]

LDoc can process Moonscript files:

http://stevedonovan.github.io/ldoc/examples/list.moon.html

But you will have to write the comments yourself - in particular, if you want parameter type info you will have to annotate those comments!

On Sun, Jun 19, 2016 at 9:57 PM, Fox notifications@github.com wrote:

Has there been any more work on something like this or ideas or anything?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/leafo/moonscript/issues/105#issuecomment-227017068, or mute the thread https://github.com/notifications/unsubscribe/AANXDeJSZd16v3G6esHeHBYnBuU45Kd-ks5qNZ8pgaJpZM4A33lY .

[TangentFoxy]

Thank you! I've been successfully updating my project to use LDoc, and this is so much nicer than writing everything by hand.