Migrate manual docs in `lua.txt` into doxygen.

[lewis6991]

Half of lua.txt is manually maintained, and half of it is generated via doxygen. More modules should be generated with the source docs moving to the implementation.

This will further the goal of #329

Files to target:

• _meta.lua
• src/nvim/lua/*.lua (vim.diff, vim.mpack, etc)

Problems to solve

Support Lua-C functions

e.g. nlua_xdl_diff maps to vim.diff, however there is currently no (easy) way give this information to doxygen.

We can implement doxygen filter which renames the function. e.g. int nlua_xdl_diff(lua_State *lstate) -> vim.diff(a, b, opts). This will be very similar to what lua2dox.lua does currently.

Support other things than functions

lua2dox currently only supports documenting functions. We can add support so it can document any lua object for things like vim.bo, vim.g, vim.wo, etc.

[clason]

Do we need to go through doxygen? It starts sounding like our own custom translator (that only supports the features we need, not all of doxygen) would be the simpler and more maintainable option.

[lewis6991]

Either that or switch to something else.

One of the things good about doxygen is that it parses the comments pretty well into xml, which is most of the heavy lifting of gen_vimdoc.py.

[lewis6991]

I can't find any standard comment parser which would be a good alternative to doxygen. So if we want to use something else, the only Idea I have would be to create an emmy TS parser which can be injected into comments of any language. This would be a fair amount of work but would give us absolute flexibility.

[clason]

an emmy TS parser which can be injected into comments of any language.

That sounds indeed like a worthwhile and very useful alternative. Note that there is already a tree-sitter-comment parser which nvim-treesitter uses for injections and could be extended in this direction.

Specifically for Lua, there's of course https://github.com/tjdevries/tree-sitter-lua, which has an explicit focus on docgen.

(Not saying that this is the direction we should be pursuing, and now, mind you!)

[folke]

We could create runtime/lua/types.lua. This file would never actually be required. We could even add an error on top to be sure.

In that file, we can then add the manual docs from lua.txt with the correct functions.

Doing it this way, we have:

• a good way to document those functions
• lua-language-server can pick up those annotations and provide completions

Would definitely make my work with lua-dev.nvim easier.

Right now, I'm parsing the doc/lua.txt files and then regenerate the lua source automatcially. (with a lot of limitations)

See the generated file below. Could be a good starting point, since it literally includes the docs from doc/lua.txt.

These are only the functions that don't actually exist in the lua code, so the manual ones from doc/lua.txt

types/lua.lua

```lua ---@meta --# selene: allow(unused_variable) ---@diagnostic disable: unused-local -- Execute Vim script commands. -- -- Note that `vim.cmd` can be indexed with a command name to return a -- callable function to the command. -- -- Example: -- ```lua -- -- vim.cmd('echo 42') -- vim.cmd([[ -- augroup My_group -- autocmd! -- autocmd FileType c setlocal cindent -- augroup END -- ]]) -- -- -- Ex command :echo "foo" -- -- Note string literals need to be double quoted. -- vim.cmd('echo "foo"') -- vim.cmd { cmd = 'echo', args = { '"foo"' } } -- vim.cmd.echo({ args = { '"foo"' } }) -- vim.cmd.echo('"foo"') -- -- -- Ex command :write! myfile.txt -- vim.cmd('write! myfile.txt') -- vim.cmd { cmd = 'write', args = { "myfile.txt" }, bang = true } -- vim.cmd.write { args = { "myfile.txt" }, bang = true } -- vim.cmd.write { "myfile.txt", bang = true } -- -- -- Ex command :colorscheme blue -- vim.cmd('colorscheme blue') -- vim.cmd.colorscheme('blue') -- -- ``` --- @see |ex-cmd-index| --- @param command any # string|table Command(s) to execute. If a string, executes -- multiple lines of Vim script at once. In this case, it is -- an alias to |nvim_exec()|, where `output` is set to false. -- Thus it works identical to |:source|. If a table, executes -- a single command. In this case, it is an alias to -- |nvim_cmd()| where `opts` is empty. function vim.cmd(command) end -- Return a human-readable representation of the given object. --- @see https://github.com/kikito/inspect.lua --- @see https://github.com/mpeterv/vinspect --- @param options table function vim.inspect(object, options) end -- Invokes |vim-function| or |user-function| {func} with arguments {...}. -- See also |vim.fn|. -- Equivalent to: > -- vim.fn[func]({...}) --- @param func fun() function vim.call(func, ...) end -- Run diff on strings {a} and {b}. Any indices returned by this function, -- either directly or via callback arguments, are 1-based. -- -- Examples: > -- -- vim.diff('a\n', 'b\nc\n') -- --> -- @@ -1 +1,2 @@ -- -a -- +b -- +c -- -- vim.diff('a\n', 'b\nc\n', {result_type = 'indices'}) -- --> -- { -- {1, 1, 1, 2} -- } -- < -- Parameters: ~ -- {a} First string to compare -- {b} Second string to compare -- {opts} Optional parameters: -- • `on_hunk` (callback): -- Invoked for each hunk in the diff. Return a negative number -- to cancel the callback for any remaining hunks. -- Args: -- • `start_a` (integer): Start line of hunk in {a}. -- • `count_a` (integer): Hunk size in {a}. -- • `start_b` (integer): Start line of hunk in {b}. -- • `count_b` (integer): Hunk size in {b}. -- • `result_type` (string): Form of the returned diff: -- • "unified": (default) String in unified format. -- • "indices": Array of hunk locations. -- Note: This option is ignored if `on_hunk` is used. -- • `algorithm` (string): -- Diff algorithm to use. Values: -- • "myers" the default algorithm -- • "minimal" spend extra time to generate the -- smallest possible diff -- • "patience" patience diff algorithm -- • "histogram" histogram diff algorithm -- • `ctxlen` (integer): Context length -- • `interhunkctxlen` (integer): -- Inter hunk context length -- • `ignore_whitespace` (boolean): -- Ignore whitespace -- • `ignore_whitespace_change` (boolean): -- Ignore whitespace change -- • `ignore_whitespace_change_at_eol` (boolean) -- Ignore whitespace change at end-of-line. -- • `ignore_cr_at_eol` (boolean) -- Ignore carriage return at end-of-line -- • `ignore_blank_lines` (boolean) -- Ignore blank lines -- • `indent_heuristic` (boolean): -- Use the indent heuristic for the internal -- diff library. -- -- Return: ~ -- See {opts.result_type}. nil if {opts.on_hunk} is given. --- @param opts table function vim.diff(a, b, opts) end -- The result is a String, which is the text {str} converted from -- encoding {from} to encoding {to}. When the conversion fails `nil` is -- returned. When some characters could not be converted they -- are replaced with "?". -- The encoding names are whatever the iconv() library function -- can accept, see ":Man 3 iconv". -- -- Parameters: ~ -- {str} (string) Text to convert -- {from} (string) Encoding of {str} -- {to} (string) Target encoding -- -- Returns: ~ -- Converted string if conversion succeeds, `nil` otherwise. --- @param str string --- @param from number --- @param to number --- @param opts? table function vim.iconv(str, from, to, opts) end -- Returns true if the code is executing as part of a "fast" event handler, -- where most of the API is disabled. These are low-level events (e.g. -- |lua-loop-callbacks|) which can be invoked whenever Nvim polls for input. -- When this is `false` most API functions are callable (but may be subject -- to other restrictions such as |textlock|). function vim.in_fast_event() end -- Decodes (or "unpacks") the msgpack-encoded {str} to a Lua object. --- @param str string function vim.mpack.decode(str) end -- Encodes (or "packs") Lua object {obj} as msgpack in a Lua string. function vim.mpack.encode(obj) end -- Parse the Vim regex {re} and return a regex object. Regexes are "magic" -- and case-sensitive by default, regardless of 'magic' and 'ignorecase'. -- They can be controlled with flags, see |/magic| and |/ignorecase|. function vim.regex(re) end -- Sends {event} to {channel} via |RPC| and returns immediately. If {channel} -- is 0, the event is broadcast to all channels. -- -- This function also works in a fast callback |lua-loop-callbacks|. --- @param args? table function vim.rpcnotify(channel, method, args) end -- Sends a request to {channel} to invoke {method} via |RPC| and blocks until -- a response is received. -- -- Note: NIL values as part of the return value is represented as |vim.NIL| -- special value --- @param args? table function vim.rpcrequest(channel, method, args) end -- Schedules {callback} to be invoked soon by the main event-loop. Useful -- to avoid |textlock| or other temporary restrictions. --- @param callback fun() function vim.schedule(callback) end -- Check {str} for spelling errors. Similar to the Vimscript function -- |spellbadword()|. -- -- Note: The behaviour of this function is dependent on: 'spelllang', -- 'spellfile', 'spellcapcheck' and 'spelloptions' which can all be local to -- the buffer. Consider calling this with |nvim_buf_call()|. -- -- Example: > -- -- vim.spell.check("the quik brown fox") -- --> -- { -- {'quik', 'bad', 4} -- } -- < -- Parameters: ~ -- {str} String to spell check. -- -- Return: ~ -- List of tuples with three items: -- - The badly spelled word. -- - The type of the spelling error: -- "bad" spelling mistake -- "rare" rare word -- "local" word only valid in another region -- "caps" word should start with Capital -- - The position in {str} where the word begins. --- @param str string function vim.spell.check(str) end -- Convert UTF-32 or UTF-16 {index} to byte index. If {use_utf16} is not -- supplied, it defaults to false (use UTF-32). Returns the byte index. -- -- Invalid UTF-8 and NUL is treated like by |vim.str_byteindex()|. -- An {index} in the middle of a UTF-16 sequence is rounded upwards to -- the end of that sequence. --- @param str string --- @param index number --- @param use_utf16? any function vim.str_byteindex(str, index, use_utf16) end -- Convert byte index to UTF-32 and UTF-16 indices. If {index} is not -- supplied, the length of the string is used. All indices are zero-based. -- Returns two values: the UTF-32 and UTF-16 indices respectively. -- -- Embedded NUL bytes are treated as terminating the string. Invalid UTF-8 -- bytes, and embedded surrogates are counted as one code point each. An -- {index} in the middle of a UTF-8 sequence is rounded upwards to the end of -- that sequence. --- @param str string --- @param index? number function vim.str_utfindex(str, index) end -- Compares strings case-insensitively. Returns 0, 1 or -1 if strings are -- equal, {a} is greater than {b} or {a} is lesser than {b}, respectively. function vim.stricmp(a, b) end -- Attach to ui events, similar to |nvim_ui_attach()| but receive events -- as lua callback. Can be used to implement screen elements like -- popupmenu or message handling in lua. -- -- {options} should be a dictionary-like table, where `ext_...` options should -- be set to true to receive events for the respective external element. -- -- {callback} receives event name plus additional parameters. See |ui-popupmenu| -- and the sections below for event format for respective events. -- -- WARNING: This api is considered experimental. Usability will vary for -- different screen elements. In particular `ext_messages` behavior is subject -- to further changes and usability improvements. This is expected to be -- used to handle messages when setting 'cmdheight' to zero (which is -- likewise experimental). -- -- Example (stub for a |ui-popupmenu| implementation): > -- -- ns = vim.api.nvim_create_namespace('my_fancy_pum') -- -- vim.ui_attach(ns, {ext_popupmenu=true}, function(event, ...) -- if event == "popupmenu_show" then -- local items, selected, row, col, grid = ... -- print("display pum ", #items) -- elseif event == "popupmenu_select" then -- local selected = ... -- print("selected", selected) -- elseif event == "popupmenu_hide" then -- print("FIN") -- end -- end) --- @param ns number --- @param options table --- @param callback fun() function vim.ui_attach(ns, options, callback) end -- Detach a callback previously attached with |vim.ui_attach()| for the -- given namespace {ns}. --- @param ns number function vim.ui_detach(ns) end -- Gets the version of the current Nvim build. function vim.version() end -- Wait for {time} in milliseconds until {callback} returns `true`. -- -- Executes {callback} immediately and at approximately {interval} -- milliseconds (default 200). Nvim still processes other events during -- this time. -- -- Parameters: ~ -- {time} Number of milliseconds to wait -- {callback} Optional callback. Waits until {callback} returns true -- {interval} (Approximate) number of milliseconds to wait between polls -- {fast_only} If true, only |api-fast| events will be processed. -- If called from while in an |api-fast| event, will -- automatically be set to `true`. -- -- Returns: ~ -- If {callback} returns `true` during the {time}: -- `true, nil` -- -- If {callback} never returns `true` during the {time}: -- `false, -1` -- -- If {callback} is interrupted during the {time}: -- `false, -2` -- -- If {callback} errors, the error is raised. -- -- Examples: > -- -- --- -- -- Wait for 100 ms, allowing other events to process -- vim.wait(100, function() end) -- -- --- -- -- Wait for 100 ms or until global variable set. -- vim.wait(100, function() return vim.g.waiting_for_var end) -- -- --- -- -- Wait for 1 second or until global variable set, checking every ~500 ms -- vim.wait(1000, function() return vim.g.waiting_for_var end, 500) -- -- --- -- -- Schedule a function to set a value in 100ms -- vim.defer_fn(function() vim.g.timer_result = true end, 100) -- -- -- Would wait ten seconds if results blocked. Actually only waits 100 ms -- if vim.wait(10000, function() return vim.g.timer_result end) then -- print('Only waiting a little bit of time!') -- end -- < --- @param callback? fun() --- @param interval? any --- @param fast_only? any function vim.wait(time, callback, interval, fast_only) end ```

Similar files are created for vim.fn, vim.api, vim options, vim.uv

[justinmk]

Avoiding doxygen is orthogonal to the original topic here.

Right now, I'm parsing the doc/lua.txt files and then regenerate the lua source automatcially. (with a lot of limitations)

• fwiw, this may be useful: https://github.com/neovim/tree-sitter-vimdoc
  • this is used by gen_help_html.lua which generates the HTML docs: https://neovim.io/doc/user

See the generated file below. Could be a good starting point, since it literally includes the docs from doc/lua.txt.

yeah, that is similar to this dummy function for the similar purpose: https://github.com/neovim/neovim/blob/26c653718097955dc4dfbeb45ab602c8dbe9dea5/runtime/lua/vim/shared.lua#L20

seems like a fine way to go. It keeps the docs well-formatted. Manually writing :help docs should be avoided for APIs. And :help as a format should "wither on the vine" as we increment towards https://github.com/neovim/neovim/issues/329 .

[yochem]

I would like to have a go with this. Couple questions before I start:

1. Will we use @folke's approach?
2. do we want to split vim.fn, vim.api, vim options, vim.uv in multiple files or put it all in one (types.lua)?
3. Is it the goal to move all documentation in runtime/doc/lua.txt to .lua files? Such that lua.txt can be completely generated?

I now see that neodev.nvim basically already has these lua files. Would it be possible to use these?

[lewis6991]

We need to develop a custom parser so we can extract the documentation from source. AFAIK there aren't any good open source ones so I think we need a custom one.

The current approach uses doxygen + gen_help.py + lua2dox all of which have limitations.

I'm currently working on an lpeg parser which can replace much of the fragile pattern logic in lua2dox. Once that's in place, we'd be in a much stronger position to heavily refactor and simplify the doxygen + gen_help.py flow (and maybe even replace it), as it would be easy enough to extend that to C.

As mentioned above, the other alternative is to use a treesitter based solution, however I think that's a little less straight forward and might be less flexible.

I now see that neodev.nvim basically already has these lua files. Would it be possible to use these?

I not sure how usable that is, some of it might be.

[numToStr]

fwiw I already created an emmylua parser for lemmy-help (vimdoc generator). IDK how useful it is to nvim-core in its current state, but I am willing to work on it to adapt to the needs.

[lewis6991]

I did take a look at that, however I don't think we can use it since it's implemented in rust.

I'm currently aiming for an lpeg based solution since lpeg is pretty good and flexible and we're already using it for the API dispatch.

[folke]

@lewis6991 there's also the new https://github.com/amaanq/tree-sitter-luadoc by @amaanq.

I'm pretty sure it has everything that's needed for doc gen

[lewis6991]

We also need to support C

[amaanq]

Is it still doxygen? I can write a parser for that

[lewis6991]

Done in #24363.

For now we will just manually maintain files in runtime/lua/vim/_meta.