Stablize Rust-Native Completion Engine Tracking Issue

[epage]

Maintainer's notes:

Remaining work for feature parity

• [x] Communicate with bash (#3656)
• [x] Complete path ValueHints (#3656)
• [x] Complete possible values (#3656)
• [x] Complete short flags (#3656)
• [x] Complete long flags (#3656)
• [x] Complete subcommands (#3656)
• [ ] Communicate with zsh (#3916)
• [ ] Communicate with fish (#3917)
• [ ] Communicate with Powershell (#3918)
• [ ] Communicate with Elvish (#3919)
• [ ] Flag values (#3920)
• [x] Multiple values (#3921)
• [x] Hidden (#3951)
• [ ] Handling of space (#5587)
• [ ] Fix handling of OsString to preserve non-UTF8 paths
• [x] Document that stdout should not be used before getting to completion processing in case the completions are requested (#5643)
• [ ] FIGNORE support?
• [x] Fulfill #2488 for these new completions. Ideally, we have the registration script sourced on-demand in the docs so that it is "auto-updating" with the users tool
• [x] #3930
• [ ] Verify each shell's behavior with quoting to see if we need to add quoting to what we output

Non-blocking work

• Smarter selection of what to show (#5058)
• Delimited values (#3922)
• is_require_equal_set support (#3923)
• last support
• trailing_var_arg support

• 5284 (doesn't appear supported by static completions)

• Removing options based on conflicts (self and others)
• Any App or Arg setting not currently handled by the static completions
  • Priority to value delimiter and requires equals
• Complete more ValueHints
• Figure out where this will live as the dependency requires are dramatically different than the static completions and this shouldn't be behind a feature flag for forever
• Custom help messages (#5063)
• Completing multiple path elements at once (#5279)

Design considerations

• Make it easy to use for the natively supported shells but allow other shells to be supported

Prior art

• https://github.com/pacak/bpaf
• https://pypi.org/project/argcomplete
• https://github.com/rsteube/carapace
• https://github.com/microsoft/inshellisense
• https://github.com/sigoden/argc-completions
• https://click.palletsprojects.com/en/8.1.x/shell-completion/

---

3022 was a tipping point for me in realizing that maybe our current approach to completions doesn't work. We effectively have to implement a mostly-untested parser within each shell. Examples of other problems that seem to stem from this:

• https://github.com/clap-rs/clap/issues/2729
• https://github.com/clap-rs/clap/issues/2750
• https://github.com/clap-rs/clap/issues/2145
• https://github.com/clap-rs/clap/issues/1822
• https://github.com/clap-rs/clap/issues/1764

If we take the approach of argcomplete where we do the parsing in our core code, rather than in each completion script, this will help us share parsing logic between shell, share some or all parsing logic with clap itself, and make a subset of the logic more testable.

We also need to decide whether to morph the existing parser into supporting this or create a custom parser (since the needs are pretty special). If we do a custom parser, we should probably do https://github.com/clap-rs/clap/issues/2915 first so we can reuse lexing logic between clap and the completion generation. I've also been considering https://github.com/clap-rs/clap/issues/2912 which would allow reusing the completion logic with any CLI parser.

[epage]

Bash's expectations

Inputs for -F and -C:

• COMP_LINE: The current command line. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_POINT: The index of the current cursor position relative to the beginning of the current command. If the current cursor position is at the end of the current command, the value of this variable is equal to ${#COMP_LINE}. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_KEY: The key (or final key of a key sequence) used to invoke the current completion function.
• COMP_TYPE: Set to an integer value corresponding to the type of completion attempted that caused a completion function to be called: TAB, for normal completion, ‘?’, for listing completions after successive tabs, ‘!’, for listing alternatives on partial word completion, ‘@’, to list completions if the word is not unmodified, or ‘%’, for menu completion. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• 1: name of the command whose arguments are being completed
• 2: word being completed
• 3: word preceding the word being completed on the current command line

Inputs for -F:

• COMP_WORDS, COMP_CWORD when used with -F

Output for -C:

• print a list of completions, one per line, to the standard output. Backslash may be used to escape a newline, if necessary.

Output for -F:

• It must put the possible completions in the COMPREPLY array variable, one per array element.

See

• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion
• https://www.gnu.org/software/bash/manual/html_node/A-Programmable-Completion-Example.html#A-Programmable-Completion-Example
• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins
• https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html

[epage]

argcomplete emulates bash's interface in fish by

    set -x COMP_LINE (commandline -p)
    set -x COMP_POINT (string length (commandline -cp))
    set -x COMP_TYPE

and then just registers that function

https://github.com/kislyuk/argcomplete/blob/develop/argcomplete/shell_integration.py#L60

[epage]

Value hints are supported on

• zsh
• fish

Tooltips are supported on

• zsh
• powershell
• elvish
• fish

We should make sure we don't lose these features as part of this transition, ie we shouldn't drop to the lowest common denominator.

[epage]

For Powershell

Register-ArgumentCompleter
        -CommandName <String[]>
        -ScriptBlock <ScriptBlock>
        [-Native]
        [<CommonParameters>]

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/register-argumentcompleter?view=powershell-7.2

The block receives

When you specify the Native parameter, the script block must take the following parameters in the specified order. The names of the parameters aren't important because PowerShell passes in the values by position.

• $wordToComplete (Position 0) - This parameter is set to value the user has provided before they pressed Tab. Your script block should use this value to determine tab completion values.
• $commandAst (Position 1) - This parameter is set to the Abstract Syntax Tree (AST) for the current input line. For more information, see Ast Class.
• $cursorPosition (Position 2) - This parameter is set to the position of the cursor when the user pressed Tab.

The block provides CompletionResult

The CompletionResult object allows you to provide additional details to each returned value:

• completionText (String) - The text to be used as the auto completion result. This is the value sent to the command.
• listItemText (String) - The text to be displayed in a list, such as when the user presses Ctrl+Space. This is used for display only and is not passed to the command when selected.
• resultType (CompletionResultType) - The type of completion result.
• toolTip (String) - The text for the tooltip with details to be displayed about the object. This is visible when the user selects an item after pressing Ctrl+Space.

So it seems like Powershell can fit within rust-driven completions and provide the full feature set.

[epage]

I'm starting small and prototyping for just bash support. Looking at argcomplete, it seems they had to use their own shlex implementation. Hoping we can avoid that. The first step is https://github.com/comex/rust-shlex/issues/12.

[happenslol]

Hi, I'm interested in helping with this this I'm developing a few personal tools using clap that would hugely benefit from dynamic completions. Is there any good issues that nobody is working on you could point me towards?

[epage]

@happenslol Thanks!

Anything unchecked in the "Remaining work" section is up for grabs; just post here that you are getting started on it. The highest priority is the support for each shell as that will help gauge the feasibility and provide feedback on the API. The rest is flushing out parsing implementation.

I should split those out into individual issues to make it easier for people to coordinate on those (and to add bounties) but I probably won't have time for another week or so.

[happenslol]

Alright, had a busy week, so I'm only getting back around to this now.

I've looked at the work done in #3656, and it looks like the next steps to add zsh and fish support would be to pull some of the functionality out of clap_complete::dynamic::bash, and adapt it to the other shells to provide the same functionality. Am I correct in assuming that?

I'd look at argcomplete for the other implementations here, since the current one also seems to be coming from there.

[epage]

@happenslol Yes, that is correct. I'd like to focus on shell support initially as that is where the most unknowns exist

[happenslol]

Started work here. I'm looking at cobra for zsh completions since argcomplete basically takes the easy way out and tells people to enable bash completion support for zsh :sweat_smile:

[epage]

Oh if cobra is doing the same type of dynamic completions as argcomplete, that is great! That'll serve as a much better example!

[happenslol]

Yeah, the cobra code is very well documented and easy to adapt.

One thing I was wondering about is the way options like space/no-space are passed to the completion command. cobra deals with this in a pretty involved way by encoding all options (compile-time and run-time) into a bitfield, and then parsing that bitfield in their completion script. It wouldn't be too hard to adapt this, but I feel like there isn't a clear definition of what options we even want to offer and the completion output would have to be changed, so that's probably better for another PR.

For a start, I would try to keep the features that are implemented for bash.

Should I open a new issue for zsh communication for questions like these, or is there some chat/forum where we could discuss them?

[epage]

Yeah, the cobra code is very well documented and easy to adapt.

One thing I was wondering about is the way options like space/no-space are passed to the completion command. cobra deals with this in a pretty involved way by encoding all options (compile-time and run-time) into a bitfield, and then parsing that bitfield in their completion script. It wouldn't be too hard to adapt this, but I feel like there isn't a clear definition of what options we even want to offer and the completion output would have to be changed, so that's probably better for another PR.

For a start, I would try to keep the features that are implemented for bash.

Should I open a new issue for zsh communication for questions like these, or is there some chat/forum where we could discuss them?

I created https://github.com/clap-rs/clap/issues/3916. We can also talk on rust-lang zulip on the WG-CLI stream

[blaggacao]

I want to apport a data point:

• https://github.com/rsteube/carapace

Implements cobra completion with advanced things like a fs cache for dynamic completion querys (e.g. with provenience from remote APIs) for an x amount of seconds.

Imagine a aws CLI wrapper that interacts with your custom infrastructure and things like it.