Maintain documentation in a machine readable format

[lewis6991]

Problem

Neovim currently translates the markdown documentation from this repo into its own vim-help format. This is currently done manually when we update Luv.

Further to this https://github.com/folke/neodev.nvim parses this documentation (in vim-help format) to generate type annotations for luv (for use with Lua-LSP).

Request

To significantly simplify both of these workflows, can docs.md be ported to a machine readable format (I propose) Lua, from which docs.md can be generated from a small auxiliary script?

I am happy to implement this myself. Just want to know in advance if this is an acceptable solution?

[squeek502]

Sounds good to me!

[Bilal2453]

So I have actually been working on this for a while now. My idea was to fully type Luvit out (including Luv) in the EmmyLua annotation, then parse that into JSON, then from there we can use that JSON for whatever else (including regenerating the EmmyLua annotations); Kinda the bootstrapping approach, but I thought that would make the most sense for me.

Currently I just need to choose the JSON structure and write the parser for it, from there we can write a markdown generator that produces something similar to the current docs. This will end up generating about the same docs we have with the exception of the introduction, since the JSON will only contain actual data types and functions with their descriptions.

The data includes: classes, methods, descriptions for methods and classes (identical to the one we have in docs), arguments, returns, returns that cannot be discarded[1], constants, and a lot more additional data structures that can be returned by Libuv (like for example, the exact possible returns of req_get_type), or accepted by it (like the exact protocols accepted by uv.socketpair).

[lewis6991]

My idea was to not write a parser at all (and hence no dependency) and just keep the structure in source in Lua. From that we can render the annotations and documentation.

If we want to go to the parser root then I'd probably just implement a treesitter parser since that would be useful in other projects.

Or alternatively and lpeg based parser which might be simpler.

[Bilal2453]

That is an interesting idea. So basically, use the Lua structure that hold the docs to generate the markdown for the docs?

[lewis6991]

That is an interesting idea. So basically, use the Lua structure that hold the docs to generate the markdown for the docs?

Yep. Basically keep the source in the simplest and easiest to process format, from which we render it into more complex formats.

[Bilal2453]

I was actually going the LPeg parser route here. And the end result would be a JSON that then could be parsed into whatever. But having this in Lua directly would indeed remove the JSON dependency which would be neat.

Was there anything preventing you from already doing that? I mean you got the annotation thing for the LLS for a while now.

I guess I will instead be parsing the LLS annotations into Lua structures then, shouldn't be that more hard compared to JSON I believe. My end goal was effectively re-documenting Luvit, which already has its docs in JSON but it depends on Nodejs, I preferred redoing it all over working with that lol. (And it already was a lot of reading Luvit source code to document types, which have not been documented in the original docs)

[lewis6991]

When I raised the issue I wasn't aware the annotations we're already in LLS so that reduced the priority of this. However we still want this so we can render the docs in vimhelp format.

[SinisterRectus]

I don't care what the format is as long as it's eventually human readable with minimal effort. A Lua format sounds fine to me.

[Bilal2453]

When I raised the issue I wasn't aware the annotations we're already in LLS so that reduced the priority of this. However we still want this so we can render the docs in vimhelp format.

I see, that sounds good to me then. The way I am imagining this currently is we have a docs directory which holds on data.lua, this files simply returns a table that hold all the details one need to generate the docs, and a generate-md.lua that makes use of data.lua to generate the final markdown docs. The final docs.md can reside there too, perhaps in a results sub-directory.

This way people can easily make other scripts to generate different types of output. Also makes it easy to provide a script that scrapes Libuv docs for any updates saving manual work (since the docs were originally generated from there it seems)

[Bilal2453]

What Lua structure/format do you have in mind for this data.lua?

Data I can provide:

• classes.
• methods.
• descriptions of classes and methods.
• methods arguments and returns.
• what returns cannot be ignored.
• eventually, the fields of any table data structure that can be accepted or returned
• any constant value that can be accepted or returned.
• also currently working on: where each section was defined in Luv C code (e.g. documenting that new_tcp is defined in tcp.c.

[lewis6991]

I haven't put any thought into the structure yet, but basically the bare minimum to generate what we need: annotations+markdown.

If you want to take over this then feel free.

[Bilal2453]

Progress update: Just started actually working on this, will try to get something bootstrapped over at https://github.com/Bilal2453/luv-docgen, once that is done will be contributing something like docs/data.lua and a docs/generate-markdown.lua over here.