Improvement suggestion: Dynamic completion

[avshenuk]

I have an idea of dynamic completion in case if your completion candidates depend on the context of the overall command and more specifically on all previously typed params/options. Let's say you want to build a kind of FTP client CLI. And you want to traverse folders one by one until you get to the proper file (the same way a generic filesystem works) and you want them to be autocompleted (of course). So that any next folders/files to show depend on the folder you've already typed.

So at the very basis the idea is to reuse the existing command as an auto-completer which the bash completion script will call with all existing parameters in the COMP_WORDS variable. To be precise, to have a possibility to write a special class (serving as completionCandidates field that we already have) but instead it gets your own command as a context with potentially some of the fields populated and based on that produces a list of candidates.

Also we'd need a boolean flag on @Command annotation to say "I want this magic super-duper dynamic autocompletion" and then internally we generate a special command that will accept all same params as our command (or just an args array because we don't care), populate our command with those params (just call .parse) and send our command as a context to our special autocompletion class that in its turn will produce a list of completion candidates and return back (via System.out) to Bash.

The question is: do we have some kind of autogeneration of Java classes? And how could we connect that special command with the whole command hierarchy? Like, could we have a hidden subcommand of the main/sub command that accepts a raw args array?

[remkop]

I understand the use case, but to be honest, I still have trouble seeing how the bash completion logic would be able to invoke a java program to generate completions.

About the special mode to dynamically generate completions, I believe a starting point for something like this is already available: picocli-3.5 added acomplete function to AutoComplete. This is used to implement auto-completion for picocli commands in JLine 2.

I am about to release a new module with components and documentation for combining JLine with picocli. Initially this will have a JLine 2 completer based on the CommandSpec. JLine completers can be very powerful. It should be possible to do what you describe in a custom Completer. It would also circumvent the problem of invoking java from the bash completion function (because we stay inside the JLine shell java program).

[avshenuk]

Yes, I really like the idea of integration of JLine with picocli. But the only downside which is pretty critical to me right now is a possibility to chain/pipe my tool with other system tools, e.g. jq, which in case of JSON processing adds a lot of stuff on top of my own stuff.
I've tried searching for a simple way of integrating JLine with system tools but seems it's not just like that (probably it can be implemented using ProcessBuilder etc. but that's too much effort as for me)...

As for the "how the bash logic would call a Java program" I though it would simply rely on the name of the command (and assume you have a Bash script/alias invoking your command). A bit shaking solution though...

[remkop]

Sounds like JLine is not the best solution for your use case.

As for the "how the bash logic would call a Java program" I though it would simply rely on the name of the command (and assume you have a Bash script/alias invoking your command).

Perhaps I misunderstood. I thought you were thinking to invoke a Java program from inside the bash completion logic to dynamically generate additional completion candidates. (So this is before the actual command is invoked.) I don't know if it possible for an external program to provide completion candidates to the bash complete built-in.

[avshenuk]

Hm, so assuming we have a command called "mycomm". And if we generate a subcommand that can be called like "mycomm complete".
Then the Bash script that we currently generate could include before calling COMPREPLY=( $( compgen -W "${candidates}" -- ${CURR_WORD} ) ) something like: candidates=$(mycomm complete "${COMP_WORDS[@]}"), which will call our special command that in its turn will parse the args in scope of our mycomm and generate the candidates based on that.
The only thing is that mycomm must be globally available in Bash (which is probably the first thing you do for easy access to your command, not during development though).
The convenience I see here is that you don't need to:

1. Write the completion logic completely in Bash (what I'm currently doing and it's feasible in case you have some ready-made tools that can help you with that, but if you have everything in Java, you'd need the step 2.)
2. Write a new separate completion command and Bash script for it that you'll need to place globally alongside the Bash script that we already generate. And the issue here is that in our generated script you need to know where each of the params you're interested in is located. Let's say your command accepts the following: country, city, first name and last name. And you want to autocomplete the city. You write a brand new autocompletion command of the format complete <country>. Now you want to call it from the Bash script. But you have only a COMP_WORDS array which have absolutely all the args including first name and last name (which can be options intermixed with params). So you'll need to parse that array and find out where the country is. Why should you do that if you already have the main command that accepts absolutely the same set of arguments? So from the point of reusing the existing code I thought this could be useful...

[jvassev]

This feature would be great for usability. For example kubectl get namespace def<TAB> would suggest a list of namespaces available in the current settings that start with "def".

However, it seems the picocli would need a lot more metadata to make this feature "out of the box". In kubectl the completion is really "hardcoded" here: https://github.com/kubernetes/kubernetes/blob/932e657d5d0e98624fb9907ea34a55ac604253d3/pkg/kubectl/cmd/cmd.go#L103

To avoid this coupling I am imagining this interface for "suggestions".

interface Suggester {
       // somehow communicate where exactly on the line <TAB> is pressed
    String[] suggest(Spec partiallyParsedSpec, String currentToken, Object cursorPosition);
}

@Command(suggester=MySuggester.class)
class RootCmd implements Runnable {
    // ...
}

@Command()
class GetNamespace {
    @Parameter("namespace", suggestable=true)
}

If the suggester is set at the root command, then an internal, hidden subcommand (for example __suggest) would be added to the Root command. It would be invoked whenever the completion logic would detect a possible "suggestion" scenario, namely when a @ Parameter or an @ Option is marked as suggestable.

Going back to the kubectl example, when TAB is pressed in kubectl get namespace kube-sys<TAB>, the bash completion script would need to call the app (more or less) in this way:

kubectl __suggest --current-word=kube-sys -- get namespace kube-sys. Internally, the Suggester interface would be invoked with a Spec corresponding to the GetNamespace class and currentWord = "kube-sys".

Presumably, the Suggester implementation would fetch all namespaces remotely and return a string list containing only the namespaces starting with "kube-sys".

[avshenuk]

Agree :) Except that probably the "current-word" is always the last argument in the list so no need to pass it. At least the internal __suggest command can infer it and find a proper suggestable field.

[avshenuk]

Ah, no. You're right. It's needed because an empty last arg would not be handled by the command.

[remkop]

Interesting idea to have a hidden subcommand to generate completion candidates.

I hope that it won't be necessary to introduce new API for this. Please note that a lot of what is being suggested already exists:

• take a look at the complete method in AutoComplete. It generates completion candidates for partial command line input. The hidden subcommand could invoke this method to generate candidates. One question though: how would the subcommand (being Java) return these candidates to the bash function? Is it enough to print the resulting candidates to System.out or would that interfere with the bash completion function? (Something to investigate...)
• note that completionCandidates is an Iterable<String>; this is just an interface and there is nothing that prevents an implementation from dynamically adjusting its result based on other attributes of the command. (See this completion in the Micronaut CLI as an example of dynamically generating candidates.)

I think this is definitely an innovative and interesting idea and worth pursuing further. Is anyone interesting in creating a POC implementation?

My initial concern was that the resulting setup may be a bit fragile: the completion logic will now depend on the mycomm command to correctly invoke java with the right classpath etc. So when an application author writes an application and distributes it to end users, the app author needs to make sure that the end users have a mycomm script that works correctly in their environment. I was worried that the installation for end users may become more complex but perhaps this is also something that can be investigated in a POC.

[avshenuk]

Yeah, I may start from a prototype and think about the drawback you mentioned... Maybe we can infer the correct script name in some way? Like the first argument passed to the completion script may be a proper command path that we may try to use...

[jvassev]

The moment the completion script is generated we can use this invocation to infer which jvm and how it was invoked using the https://docs.oracle.com/javase/7/docs/api/index.html?java/lang/management/MemoryMXBean.html. This way the classpath, any jvm opts (-Xmx etc) , system properties could be captured in a simple function:

__invoke_mycomm() {
 /path/to/java -Xmx256m -client ... # this line is built using output from JMX bean
}

__mycomm_suggest_name() {
    # same as in previous comment
    __invoke_mycomm __suggest -- "$@" 
}

Something like https://github.com/remkop/picocli/issues/456 (proposal for another built-in subcommand) would play nicely with the rest.

[dwalluck]

First, let's clarify the issue: I think that there are two different enhancements being discussed on this issue, one discussing how to generate completions for use by the program itself (such as within a Java ftp client) and one discussing completions for bash.

Clearly, certain use cases will require that completions be dynamically generated vs. statically generated at compile time. Ideally, we could have access to any value that the JVM would have access to at runtime.

For bash, is it just that you wish to provide completion candidates dynamically (at runtime) vs. the usual static generation (at compile time)? If picocli can already generate dynamic completions for a command, then this might not be necessary. But, it sounds like the feature you are asking for is the following:

You can run an external command which returns completion candidates using the compgen -C option. After the external command is invoked, the COMP_LINE, COMP_POINT, COMP_KEY, and COMP_TYPE variables are set in the environment for the completion candidate generation code to process in order to determine the correct completions to return.

The problem is that it's hard, if not impossible, to know how to invoke a java program under the correct JVM, unless bash can just invoke the program itself (say, $0 using a hidden command-line option).

Here is an example of a python program which uses an external completion program https://github.com/aws/aws-cli/blob/develop/bin/aws_completer. However, you can see it uses #!/usr/bin/env python but something like #!/usr/bin/env java usually doesn't work or can invoke a different JVM than the one we are currently running under.

[dwalluck]

To clarify a bit more: if the current auto-completion script generator evaluates the completions at the time it is run and injects the static result of this into a string, this will fail under certain conditions.

For example, the auto-completion Java code mentions in a comment supporting, say, MessageDigest values, but this is wrong to do statically. Consider that at the time you run the generator, the JVM supports SHA-512 and SHA-256, and maybe you even have a different crypto provider such as bouncycastle on your CLASSPATH. Then, your user's Java environment is changed. Maybe he now only has support only for weaker digests such as MD-5 and SHA-1. So, when he runs the static script is his "new" environment, he will get incorrect results.

[yschimke]

I'd suggest some approach for allowing hidden options e.g. --pico_complete="" or --pico_completion_script instead of additional main methods.

Particularly for commands deployed as a graal binary, it's useful to have additional hidden options over extra binaries.

[gunnarmorling]

Just chiming in to raise my voice in support for dynamic completions. The git command is a great example for this; for instance, git branch <TAB> will show all the currently existing branches. It'd be great to have this functionality in picocli.

[gunnarmorling]

For reference, I solved this for now by means of a hidden command which simply prints out a String with the completion candidates of a given context. This is invoked from the completion script via

...
local FOO_PARAM_pos_param_args=`my-binary hidden-command`
...

[yschimke]

Yep, I've taken a similar approach https://github.com/rsocket/rsocket-cli/blob/master/zsh/_rsocket-cli#L4

But I think critically, this is functionality that you need to write for bash, zsh, fish etc. And topics like caching are very relevant if it's potentially a slow command. Would be ideal to solve this once in a common place.

[minfrin]

Was evaluating picocli to see if it could do dynamic completions, and it's a showstopper if not. Most specifically, we need to do this:

custom-tool --database-connection=foo customer show --customer-id=joe[tab]

The above would cause all command line params to the left of the [tab] to be parsed, and the dynamic completion for customer-id would connect to the database defined by params on the left, and look up suitable customer-ids starting with "joe" for autocompletion.

[remkop]

Was evaluating picocli to see if it could do dynamic completions, and it's a showstopper if not. Most specifically, we need to do this:

custom-tool --database-connection=foo customer show --customer-id=joe[tab]

The above would cause all command line params to the left of the [tab] to be parsed, and the dynamic completion for customer-id would connect to the database defined by params on the left, and look up suitable customer-ids starting with "joe" for autocompletion.

@minfrin Are you looking to accomplish this for a single-shot command (so you'd have to use bash/zsh completion) or for an interactive CLI program (so you could use a custom shell, like picocli-shell-jline3)?

[minfrin]

@minfrin Are you looking to accomplish this for a single-shot command (so you'd have to use bash/zsh completion) or for an interactive CLI program (so you could use a custom shell, like picocli-shell-jline3)?

Single shot command.

It's very common to autocomplete files/directories, what we need to autocomplete are entries in a database. To make this happen, we need to be able to call code to do the autocomplete (as opposed to something hard coded like annotations), and have that code have access to the options on the command line parsed so far (one of those options is the database connection url, required for autocomplete to do anything). Obviously in a case of where the database connection url is missing/invalid, autocomplete wouldn't be expected to do anything, but that would be up to the code to do the autocomplete above to decide.

[remkop]

@minfrin I see. I will not be able to work on this myself, but I would be very interested in pull requests. Will you be able to work on this?