Generic auto-completer for chat

[ClobberXD]

Related: #6837, #5532 (apologies if there are other existing issues that I missed). PR #5532 implements a tab-autocompletion API for client-side mods, and was disapproved by two devs, but only because it relied on CSM which wasn't (isn't?) going to be developed much further, as far as I can tell.

This issue is a feature request/discussion regarding a more generic auto-completer for the chat, built into the engine. This would include registered items, player names, chat-commands, biomes, entity names, and is extensible, meaning it can support custom lists.

---

Here's my idea:

• The engine is in charge of handling two cases - player name and chat-command auto-completion. The current word would be detected, and if the first char is a /, the current word is considered to be a chat-command, otherwise, a player name.
• The only other requirements for auto-completion is within chat-commands themselves, as far as I can tell.

I've thought of two ways of implementing this.

Impl 1

For auto-completion within chat-commands (i.e. params), the chat-command definition could support a new, more "strongly-typed" way of defining params, that allow mods to optionally specify a list to associate with an individual param for auto-completion purposes. Here's an example:

minetest.register_chatcommand("give", {
    params = {
        { "<name>", "minetest.get_connected_players()" },
        { "<item>", "minetest.registered_items" },
        { "[count]" },
        { "[wear]" }
    }
})

The idea is that the second item in the ParamSpec is a string containing either the name of a table, or a function that returns a table when executed; basically, a Lua snippet. The auto-completer then searches through the indices if the table is a dictionary or a hash-table, or the values themselves, if the table is a list, and returns the closest match to the current word.

(Of course, the existing syntax params = "..." would still be retained and perfectly valid)

Here's another example to demonstrate associating custom lists with params:

minetest.register_chatcommand("locate", {
    description = "Locate team-member",
    params = {
        -- To obtain list of players in a team, use ctf.team[<team_name>].players
        { "<teammate_name>", "ctf.team[ctf.player(@name).team].players" }
    }
})

This fictional chat-command highlights the location of a specified team-member of the player. Notice this line:

        { "<teammate_name>", "ctf.team[ctf.player(@name).team].players" }

The team-name of the player is required to obtain the list of the players in their team. The code gets the team-name by using the @name token to obtain the name of the player executing the chat-command. I chose this example because I wanted to outline the importance of allowing the second string in a ParamSpec to be able to access the player's name. It allows for a metric ton of possibilities, giving complete freedom to mods to use auto-completion wherever they want. Mods can extend this even further, by using local functions to retrieve the list/table, and simply entering the function name as the second item in the ParamSpec. e.g.

function ctf.get_teammates(name)
    local team_name = ctf.player(name).team
    return ctf.team[team_name].players
end

minetest.register_chatcommand("locate", {
    params = {
        { "<teammate_name>", "ctf.get_teammates(@name)" }
    }
})

Impl 2

We can drop the idea of ParamSpec, and provide a callback field on_autocomplete in the chat-command def. This callback would be provided with the player name, complete text entered, cursor position, etc. and use the data to return a string that auto-completes the current word. Here's an example:

minetest.register_chatcommand("", {
    on_autocomplete = function(name, text, current_word, cursor_pos)
        -- Do stuff

        return string_that_replaces_current_word
    end
})

The first approach is more automatic while still being pretty flexible, in which the engine takes care handing the input text, obtaining the current word, and actually retrieving the most relevant result from the associated list in the ParamSpec. This implementation is probably the faster of the two, as everything takes place in C++, except for the small bit of Lua code in the ParamSpec that fetches the table at runtime.

The second approach is a little more manual and tedious but allows for the exact same functionality. But here, the mod has to take care of everything, except for the inputs, which are automatically provided to the callback by the engine.

---

Discussion, feedback, inputs, opinions, suggestions, or alternatives most welcome.

EDIT: These implementations only concern the chat-command param auto-completion. Apart from that, there's only player name auto-completion (already implemented) and auto-completion for the actual chat-command name, both of which can be safely handled client-side.

[rubenwardy]

I prefer the second implementation, there can be helpers server side for things like params completion

[Desour]

Are you suggesting server-side autocompletion? Wouldn't this be horrible on servers with much lag?

[ClobberXD]

Are you suggesting server-side autocompletion? Wouldn't this be horrible on servers with much lag?

Ideally, auto-completion for engine tables like minetest.registered_items would be performed client-side, while mod-provided/function-returned tables would have to be processed server-side. What I didn't mention in the first post is that only the ParamSpec implementation would be able to support client-side auto-completion, with server-side auto-completion as fallback. While it could be possible, I can't think of any straight-forward way to support client-side auto-completion for the on_autocomplete implementation. This is because it's the server that would run the on_autocomplete callback.

Regarding lag, it's up to the mods to ensure that they don't have to run intensive code to process auto-completion (fetching the appropriate table and retrieving the most-relevant result), be it within on_autocomplete or within the snippet in the ParamSpec. If the server's inherently laggy, I'm afraid there's not much we can do about it. Do you have anything in mind? I'm all ears.

@rubenwardy The advantage of the first implementation (a.k.a ParamSpec) is that the auto-completion tables (or the functions to retrieve them thereof) are individually specified for each param, meaning that the client can try and process it by itself if possible, as it would already know what the table is, without the server having to run any callbacks first. OTOH, going by the second approach, the server has to run the on_autocomplete callback to even know what the associated table is, even if it's just minetest.registered_items (which the client is already fully aware of).

[HybridDog]

I have implemented Impl 2 there: #4437 The callback is named autocompletion instead of on_autocomplete and has a first_invocation parameter, which depends on whether the player presses tab twice.

Impl 1 could be implemented on top of Impl 2 by automatically setting the on_autocomplete function to some function which uses the information from the params table; on_autocomplete would e.g. split the current parameter text to a table of strings, find the index of the current parameter, etc..

[ClobberXD]

And another PR closed because of non-existent CSM. But since the devs have ruled out the possiblity CSM any time soon, would it be possible to re-open #4437? It looks like a good start to me. We can try to expand upon that later, if need be.

It does indeed look like Impl 1 can be implemented on top of #4437, but would it be possible to implement client-side auto-completion whenever applicable? For auto-completion using tables like minetest.registered_items?

[HybridDog]

[…] would it be possible to implement client-side auto-completion whenever applicable?

It could be possible but may require a lot of code changes. I think it's a detour to do something which should be implemented preferably in CSM because of flexibility; a mod may e.g. have its own autocompletion table for each player's travelnet stations. The only benefit of client-side autocompletion is a faster response time. In #4437 if the server does not respond fast enough and the player changes the chat message before receiving the autocompletion information (cursorpos and new message), the autocompletion is not performed (if I remember correctly). In my opinion, server-side autocompletion is a lot better than having no autocompletion.

[ClobberXD]

Implementing client-side auto-completion using CSM would be ideal, but that's also why #4437 was rejected (as far as I can tell) - CSM isn't being actively developed. I'm thinking we can implement client-side auto-completion in C++, and convert it into a CSM-based implementation if and when CSM is fully developed.

[paramat]

:-1: Does not seem worth the effort and code complexity. The things that are typed (registered items, player names, chat-commands, biomes, entity names) are mostly technical, are highly variable and often determined by the last few characters, so you would have to type almost all characters anyway to get the correct prediction. Typing complete words is not a problem for short chat commands in MT. I can understand it for typing large amounts of common language use on a mobile phone.

Auto-complete is only significantly practical for common language use, which cannot be done in MT as it would require a dictionary for every language.

[HybridDog]

The things that are typed […] are mostly technical, are highly variable […]

That's not a problem if it is implemented with a callback, ideally with CSM.

[…] and often determined by the last few characters, so you would have to type almost all characters anyway to get the correct prediction.

I disagree. I think many characters can be skipped when entering long chatcommands (e.g. "/teleport") or entering a player name (especially "singleplayer" in singleplayer mode). Furthermore, it can be helpful to list all alternatives by pressing tab twice.

Typing complete words is not a problem for short chat commands in MT.

Many commands are short because there's no autocompletion. When I add a chatcommand, I often use unintuitive short strings, e.g. "/lstuff" instead of "/listitems", because there's no persistent chat history and typing long words is inconvenient. Worldedit even has a worldedit_shortcommands mod to enter commands more quickly.

I can understand it for typing large amounts of common language use on a mobile phone.

Auto-complete is only significantly practical for common language use, which cannot be done in MT as it would require a dictionary for every language.

Autocompletion tasks for chat command arguments in Minetest (e.g. complete an item string) somewhat resembles Bash autocompletion (e.g. complete a file name), which is significant in my opinion. I think common language autocompletion shouldn't be implemented and may already be enabled on mobile phone screen keyboards.

-1 Does not seem worth the effort and code complexity.

Almost each time I enter a chat command I'm annoyed that I cannot use autocompletion like in Bash or other programs, so I'm still in favour of adding autocompletion. Implementing it shouldn't require much effort since there's already code for it.

I'm glad that I'm not the only person who requests and thinks about autocompletion. Thanks ClobberXD. ThobberXD.

[appgurueu]

For a good autocompletion experience, smoothness is key; a round-trip to the server may be problematic, esp. if the completion doesn't match the typed continuation (in which case the completion would have to be thrown away).

Here's my proposal: There should be both a callback and a prefix tree of completions managed by the client. The server should be able to modify the prefix tree on the client by resetting it, adding a set of new entries or removing a set of entries. This would provide a good compromise between flexible completion on the server, which requires a round trip, and instant completion on the server based on a known set of words.

Chatcommands would require no special handling; they'd just be words starting with /.

[ThePython10110]

I'm not saying that this is the best way to do it, or the way that it should be implemented for Minetest, but this is how A Certain Other Voxel Game does it (I don't know how the backend implementation works, since it's closed-source):

• When not typing a command, only player names are autocompleted.
• When typing a command, autocomplete depends on the argument (if it's an argument that accepts a block/node, only blocks/nodes will appear in the autocomplete list).
• All possible results are shown in a scrollable (with mouse or arrow keys) list.
• The pointed block/node's position is also autocompleted in commands.

I like the idea of using a ParamSpec, except for commands that can have different argument orders or subcommands (like the scoreboard command from Better Commands), where the params list isn't really enough to contain the full syntax. In that case, the ParamSpec implementation really wouldn't work at all. If the on_autocomplete implementation could somehow use CSM, that would be ideal.

[NA0341]

@appgurueu wrote:

For a good autocompletion experience, smoothness is key; a round-trip to the server may be problematic, esp. if the completion doesn't match the typed continuation (in which case the completion would have to be thrown away).

I'm not a programmer, but I "optimize" things regarding UI/UX & performance aspects. As a User, I really want to use autocomplete.

There's not only the benefit of faster typing, but also to remove the need to remember short commands for long commands you don't need that often (since they can just be auto-completed).

I also think there's no real benefit from adding Regular Language Completion ~ since a user who wants to autocomplete words wouldn't only want it in one application. So it makes no sense to not do this system-wide.

---

Suggestion for Servers: For large amounts of completions or slow networks, they could be fetched as an asset by the client.

So if a user joins a server ~ at the time the client connects (if it does already measure connection quality & speed at that time), it could use that to judge wheter to request completions right away or do that ondemand.

Additionally, the server itself knows whether it's completions are big or small, and can decide whether to serve them right away as an asset or provide them ondemand.

(I guess assets have some type of version indication and can be updated as needed.)