Completion system redesign

[notramo]

I have an idea of a new completion system that is lot more powerful than the current (and even than that of Zsh). It would be implemented fully in Elvish script.

Concept

• The user shouldn't spend time on switching between modes, so the completion should be integrated into INSERT mode: edit:insert:start-completion, edit:insert:jump-right-completion, edit:insert:trigger-completion-filter, edit:insert:exit-completion.
• Also, it should add a preview window (similar to the preview of ranger and fzf).

Advantages

• Faster typing.
• More information for the user about each completion candidate.
• More flexible and powerful completion API (more customization possibilities).
• Solving #148, #395, #337, #396 in Elvish script.
• Implementing custom commands that sorts candidates by usage frequency (similarly how the current directory history mode does).

Elvish script API for readline

It would use $edit:-dot for getting the cursor position, and $edit:current-command for getting the readline content (and inserting the completion). However, it requires more readline API features:

• 548, #549, #550 for displaying completions automatically.

• 507 for previewing the completed command (similarly to the current underlined completion preview)

Displaying the completion candidates

• Each candidate should have a tag, and a description. It should be formatted similarly how tagged paragraphs are formatted in man pages.
  Example completions:
  • options: &tag="--git" &description="list each file's Git status, if tracked"
  • commits: &tag="3ba2c02" &description="Added man page".
  • filenames: &tag="todo.txt" &description="UTF-8 Unicode text"
• It should support markup. Two ideas:
  • escape sequences
  • special syntax (my preferred solution)
    e.g. {bold}--input{-bold}={italic}file.txt{-italic}
    One advantage of this solution is that it would enable nonbreaking whitespaces: Execute {nobreak}git push origin{-nobreak} in a shell.
• Each candidate should have a preview. Only the preview of the selected item should be displayed.

• It should support sections (similarly as Zsh) to separate candidates by type.

  [local branch]
  master dev new-feature
  
  [remote branch]
  origin/master origin/dev origin/new-feature
  
  [filename]
  file1.txt file2.txt file3.txt

Reading candidates

• The edit:completion-preview-delay variable should contain the number of miliseconds before running the preview script.
• The edit:completion-preview-function variable should contain a shell function that produce the preview text. The function would have two options: &height &width.
• The shell should read candidates with a command similar to edit:-narrow-read. It would take the candidates, and a function that is launched if a candidate is accepted.

What do you think about this idea?

[notramo]

@xiaq What's your opinion on this idea? Why hasn't it even got a label? Is it a so stupid idea?

@zzamboni, @muesli, @adracea, @doppioandante, @occivink, @SolitudeSF, @tw4452852, @alexherbo2, @krader1961 I'm curious what you think also.

[zzamboni]

@notramo sorry for the delay in commenting - I have been away last week.

You ideas sound good to me, but sound more like a wish list than a concrete proposal. You seem to have some concrete ideas, but I don't fully understand how this would look in implementation. Maybe a snippet of how you would envision some moderately complex completion (say, git subcommands/options) using your proposal would help.

[xiaq]

@notramo Hi, sorry for the delay in response! I've been refactoring the editor for the past week and I tend to delay response to issues, especially bigger ones, when I am in "sprint mode". I do try to make sure that issues are triaged when not in sprint mode, but please feel free to nag me when you are seeing delay in responses.

Hopefully, the refactor work should pave the way for adding more editor features and APIs!

It is a bit hard to navigate your proposal as it contains many things. Here are my answers, which I hope address most of your proposal:

• You seem to be proposing that automatic contextual completion should be the default (or maybe only). This is an attractive idea, but I am a skeptical because some completions can be expensive:

  • Filename completions on network drives, or command completion when PATH contains a directory in a network drive;

  • Completion of package names in package managers, which likely requires querying some database.

  • We also have no control over what user-supplied completions can do, they can do arbitrarily expensive stuff.

  We can run completions asynchronously in the background, but I don't like the idea that Elvish can take up resources when the user hasn't explicitly asked to. My proposal is that Elvish exposes a user-customizable whitelist of completions that can happen automatically.

• You have proposed that the completion system should be implemented entirely in Elvish, and you have proposed many new APIs (which all make a lot of sense - thank you!). I do share the vision that the Elvish editor becomes an "editor framework" some day and all the high level stuff become doable in pure Elvish script. However, we still has a long way to go:

  • Even with all the APIs you proposed, the API interface is not sufficient to implement the completion mode in Elvish. In particular, the API provides little low-level access to the UI; you cannot control, for example, how the completion list is rendered or how the scrollbar is rendered.

  • Accurate completions require parsing Elvish code. There is no API for the Elvish parser or AST yet.

  • The Elvish interpreter is not at all optimized, so the equivalent Elvish implementation of completion mode will be much slower than the Go implementation and takes up much more memory.

  By the way, although I completely dig the idea that there should be enough APIs for the completion mode to be implementable in pure Elvish script, we will probably still be shipping a Go-implemented completion mode in the elvish binary. Go is a pretty decent language for implementing editor features, and no matter how optimized Elvish becomes it's unlikely to outperform Go code.

• You have proposed an extended API for completion candidates. This is partially also there in the edit:complex-candidate API (see doc, look for "complex candidate"). This is purely a fault of me not documenting the API well. The major thing that is now missing compared to your proposal is a mechanism for adding types to candidates, which is something I plan to do.

[notramo]

@xiaq The API functions I suggested would work on a lower layer of the completion system than you think. For example, the solution for avoiding resource expensive autocompletions should be implemented in Elvish script. The readline character input hook would be called every time the user types a character, but the completion system (the Elvish part of it) could decide if it calls the completion function or not.

Even with all the APIs you proposed, the API interface is not sufficient to implement the completion mode in Elvish. In particular, the API provides little low-level access to the UI; you cannot control, for example, how the completion list is rendered or how the scrollbar is rendered.

I think, terminal drawing shouldn't have an API:

• It would be very slow.
• Very few people would use it, so it would be an unneeded part of code that only increases complexity and bugs, and hardens maintainability. I think, if someone needs modifying it for personal tastes, the Go code should be edited.
• In the HTML front-end you could have a completion list drawn with lines, and other fancy stuff instead of characters. It could be printed even with variable width font for easier readability. This applies to the scrollbar too. Even the prompt, and the typed command could be written with variable width font, because it isn't locked into a cell table.

Instead of controlling how the editor draws to the screen, it should be controlled like: "draw a completion list: ...", "draw autosuggestion: ...", "run this command if the user press a key: ...". (This is what I proposed in the many issues I opened.) This wouldn't conflict with the web front-end.

Accurate completions require parsing Elvish code. There is no API for the Elvish parser or AST yet.

I think it isn't required for starting implementing the new completion system. It should be implemented later. I opened #592.

The Elvish interpreter is not at all optimized, so the equivalent Elvish implementation of completion mode will be much slower than the Go implementation and takes up much more memory.

Yes, but I think the completion don't require too much operation. The main performance hit is that a new array is created every time instead of modifying the current. This can be slow on big arrays, however modifying it would require nearly the same computation regardless how big is the array.

You have proposed an extended API for completion candidates. This is partially also there in the edit:complex-candidate API (see doc, look for "complex candidate").

I have know about it. The lack of documentation wasn't a big problem, it was easy to discover how it works. It lacks the following things.

• Specifying a function that is called, if the user exits completion mode after selecting a candidate. This could be used to split / from the end of directory paths, if completion mode is closed, or restarting the completion.
• Specifying the entire text displayed instead of &display-suffix, that only inserts text after the completion candidate. This could be used to display only the basename of directories/files in the completion menu.
• Inline markup. Modern command line tools (e.g. fd, exa, ripgrep) use colors for easier readability. It is very comfortable.
• Automatically organizing completion candidates into a formatted list. This line is taken from xiaq/edit.elv completion for the go command: edit:complex-candidate $text &display-suffix=' '(spaces (- 14 (wcswidth $text)))$desc
• Sections.
• Preview.

@zzamboni What type of snippet would you like? Implementation, or how a completer command would look? A completer command would be similar to the current edit:complex-candidate command, except new options would be added to it. The completion system would differ from the current in the following things:

• Automatically started for some commands.
• Autosuggestion is also managed by it.
• Better interface to the user (preview, markup, etc.)

[xiaq]

Instead of controlling how the editor draws to the screen, it should be controlled like: "draw a completion list: ...", "draw autosuggestion: ...", "run this command if the user press a key: ...". (This is what I proposed in the many issues I opened.) This wouldn't conflict with the web front-end.

That API doesn't exist yet either... :)

I think it isn't required for starting implementing the new completion system. It should be implemented later. I opened #592.

I am not sure how much of the completion mode you would like to see implementable in Elvish. If your answer is "all of it", then a full-fledged parser API is required. In #592 you are proposing an API to enable some new kinds of history-based completions, but consider what the completion mode is capable today:

• It needs to know that in echo $<Tab> you are completing a variable, in e<Tab> you are completing a command.

• It needs to know that in echo "a b c" 'x y' <Tab> the command being completed is called echo, and the existing arguments are strings a b c and x y.

(I filed #594 for that.)

Can you file another issue for adding more properties to candidates?

[notramo]

@xiaq What do you think about instead of inserting completions with edit:current-command, there would be an array of the items that should be completed by the completer function? It would contain the same strings that are now passed to the completer function as arguments, and if some of these strings are modified, the editor would know where is that item located in the readline buffer, and replace it with the modified content. It wouldn't contain e.g. variable names, so these things would be completed by the editor (same behavior as currently), and the completion function is not called when completing these items. This solution can be implemented without using a parser API, which would simplify a lot in the Elvish script completion system.

[xiaq]

@notramo, I don't understand what you are proposing. Can you give an example?

Also, I tried to reread your original post; I think most of those are now covered by existing issues. Aside from what have been mentioned, I just filed #644 to cover richer metadata in candidates.

But the existing issues may still fall short of your original vision. If there are things that haven't been covered, I recommend that you do two things:

• Break your proposal into smaller feature requests, if the existing features haven't covered them. I know this will make it harder to have a coherent picture of what we want for the completion mode; I have made some GitHub issues tags, including module:edit/completion, to help organize related issues.

• Give examples of how you think completion can better work. Ideally, come up with terminal mockups for the UI, and pseudo code for the API. I have noticed that we all have slightly different terminologies to describe UI features and it makes discussion of complex features difficult. UI mockups and API examples will make such it much easier to understand ideas.

I hope that's helpful! Now I'm going to start tagging existing issues with the new tags I just made...

[notramo]

@xiaq Currently, the completer command has a signature that specifies the variables that will store its arguments while the function is running. It is possible to specify a rest argument (with @). If only one argument is specified, and it is a rest argument, it will store all arguments passed to that function. This array (created with the rest argument) don't contains $variables, shell syntax characters ({}() etc.), and only contains the arguments of the current command (so the parser filters out the other side of a pipeline |, or next command ; etc.). In my idea, this filtered array wouldn't be passed to the completion function, instead the completion function would accept no arguments, and this array would be in the edit: namespace. It have to be writable. It would contain only the strings that would be passed to the completion function if the user hits Tab with the current cursor position. The parser would store for each element the readline buffer position where that element starts. If an element is modified in this array, the editor changes the text in the readline buffer according to the position given by the parser.

readline buffer (the cursor is at the end of the line):

~/code/neph >>> exa --git -l; kak $variable src/neph.cr src/ne│

the content of the e.g. edit:argument-list array:

[ 'kak' 'src/neph.cr' 'src/ne' ]

(so the exa command is filtered out entirely, and the $variable also) change this array:

edit:argument-list[3] = 'src/neph/parser.cr'

then the readline buffer changes (the change is inserted - we changed the last element of the array, so it is inserted at the end):

~/code/neph >>> exa --git -l; kak $variable src/neph.cr src/neph/parser.cr│

[xiaq]

@notramo It's not clear to me what advantages your proposed API has; it seems to worse than what we have now. Also, when should edit:argument-list[3] = ... be executed? The completer should propose a list of candidates, not modify the buffer directly.

[notramo]

@xiaq Sorry, I have described it with mistakes. The completion function wouldn't modify the array. The Elvish script completion system would modify it when a candidate is accepted. (The edit:argument-list[3] ... command is also called from the completion system, not the completion function.) The way how completion functions work wouldn't change, because the completion system can read out this array, and pass to the completion function as arguments. Sorry for the wrong description.

I will open new separate issues for the components.

I know my proposal is quite complex, so I have created a mind map. I hope it helps understanding the idea. The process starts from the bubble at the bottom of the image, and finishes with the lightest arrow.

[xiaq]

@notramo, I see what you are proposing now. But I don't see any advantage with this approach:

• edit: is essentially a global variable. If you make completers manipulate edit:, they are less reusable and harder to test (it is too early to talk about automated testing in Elvish now, but good practices in software engineering still apply).

• Still, I think it is too much flexibility to allow the completer to specify an arbitrary callback when a candidate gets accepted. It's better to keep the behavior of accepting a candidate uniform.

The only advantage I see is that it enables one to move more of the completion mode from Go to Elvish with relatively little new API. But that comes at the cost of a worse API for authors of completers. I would like to be able to implement completion mode in Elvish one day, but it doesn't warrant breaking other things. For most people, whether the completion mode is written in Go or Elvish should be an implementation detail.

[notramo]

@xiaq It wouldn't break existing completions. I later realized that the Elvish script completion system could work with the currently existing completion libs, without breaking compatibility, but also enables implementing more flexible completions in the same system, along with the current. If I didn't described it clearly in the previous comment: completers don't manipulate the argument array in edit:argument-list. They only outputs the candidates. The completion system would modify this array.