Added in autocomplete ability to Readline

[mrkmg]

Added in tab to auto-complete functionality to Console\Readline

When tab is pressed, the function set with ->setAutocomplete() will be called with 2 parameters. The first parameter is all the text before the currently tabbed on word. The second parameter is the currently tabbed on words. The function should return an array of all the possible auto-completions. There are 3 cases:

More than 1 result All results are displayed, then the line is auto-completed as far as possible

Exactly 1 result The line is completed

No results Nothing happens

---

Also moved the setting and restoring of the STTY to public functions so the user can set and restore as needed.

[hoaproject]

Hi,

Thank you for the pull. Sorry for the late reply, I was absent :-). Can you give me an example of a “setAutcomplete function” please? Because I have tried this one:

function ( $left, $word ) {

    return ['foobar'];
}

and all the line is replaced by foobar when pressing tab. I need to return [$word . 'foobar'] which is not very convenient.

Moreover, if the function returns more than 1 result, they are all printed which is ok, but there is no reasoning, which is the main problem of your patch. There is no autocompletion by using the current line. Your patch assumes that it is part of the developer to write such an autocompleter. The interesting thing would be to propose an simple or advanced one, no? For example, the user would declare a list of words or sentences, and when pressing tab, the autocompleter will find the next appropriate words. If many choices, then it will print all of them or it will propose a menu (which is not obvious because of multiline refreshing). The developer will focus on giving a relevant list of words, and not of writing an autocompleter, this is our concern.

What do you think?

[mrkmg]

When I wrote this, I wanted to give the developer as much freedom as possible with the autocomplete. Here are some examples of autocomplete functions that would benefit from this flexibility.

//basic autocomplete
function ex1( $left, $word ){
    $matches = array();
    $words = array('action1','action2','process1','process2','run');
    $word_length = strlen($word);
    foreach($words as $word)
        if(substr($word,0,$word_length) == $word) $matches[] = $word;
    return $matches;
}

//dynamic based on left
function ex2( $left, $word ){
    if(strlen($left)){
        $first = trim($left);
        switch($left){
            case 'action1':
                $words = array('sub1action1','sub2action1');
                break;
            case 'action2':
                $words = array('sub1action2','sub2action2');
                break;
            case 'run':
                $words = array('sub1run','sub2run');
                break;
            default:
                $words = array();
                break;
        }
    }
    else{
        $words = array('action1','action2','process1','process2','run');
    }

    $matches = array();
    $word_length = strlen($word);
    foreach($words as $word)
        if(substr($word,0,$word_length) == $word) $matches[] = $word;
    return $matches;
}

//shortcuts
function ex3( $left, $word ){
    $shortcuts = array(
        'a1'=>'action1',
        'a2'=>'action2',
        'p1'=>'process1',
        'p2'=>'process2',
        'r'=>'run'
    );
    if(isset($shortcuts[$word])) return array($shortcuts[$word]);
    else return array();
}

Here you can see in the first two examples that we are auto-completing in the classic sense. Start a word and the function return words that can "complete" the word you are typing. In the ex2, it uses the left part of the line to figure out what your next auto-complete could be. In the ex3, we are using the auto-completer to do a shortcut replacement. If you type a1 and hit tab, it will replace your a1 with action1.

I see where you are coming from though. Maybe it would be better to create a simple and advanced method. The advanced method could be what I have currently implemented. The simple method could be as simple as passing an array to setAutocomplete and using that array to auto-complete the lines. Basically, the simple implementation would be ex1 built into _bindTab. The only issue I see in a simple implementation, is how do we determine what list of words to use if we want something like ex2. Maybe the first param could be a dynamic item in a database, and the developer couldn't possibly know all the possible results beforehand, as the dataset could be huge. Lets look at this situation and see if there is a simple implementation that could handle it, as I think this would be a very common situation.

A developer has a simple script to perform actions on persons in a building. At a prompt there are 3 actions: enter, move, and leave. For enter and leave there is one parameter, a persons name. For move there are 2 parameters, the persons name and what room they are going to move to. We only want to auto-complete for rooms a person could move to, which is dependent on what room they are in. This dependence on dynamic information is what made me want to not limit the developer.

I think a simple implementation will only cover 1 use case. When you only have one list of options to choose from, always. While this may often be the case at first, like to choose enter, move, or leave it is not the case when you want to start doing anything with dynamic data, especially if that dynamic data is the determining factor for the next list of auto-complete words.

If anything I have said is unclear, or you would like more examples just let me know. I think the simple and advanced implementation is a good idea, but I think we need to think of a simple implementation that we be more powerful than just ex1

[hoaproject]

Thank you for the clarifications.

I was thinking to make several classes to represent each kind of autocompletion, such as: word, command-line etc. (I don't see more). word could fit ex1, and command-line could fit ex2 in that you need context: autocomplete command name, autocomplete a specific option etc. (please, see Zsh autocompletion for more examples). In this case, we will have several classes with an __invoke method, and a shortcut method on Hoa\Console\Readline to define the current autocompleter, such as:

$autocompleter = new Hoa\Console\Readline\Autocompleter\Word([
    'foo',
    'bar',
    'baz',
    'qux'
]);
$readline->setAutocompleter($autocompleter);

The _bindTab method will stay almost the same (the idea is good but it needs to be ameliorate to catch more cases properly).

We should start by the word autocompleter. It could autocomplete every word in a line, not only the first word: this, is part of _bindTab. So maybe there are contextes to consider: word in 1st position, 2nd position and so on, or maybe a series of potential words etc?

In all cases, we need to define an interface (even a very simple one) to enable user to make its own autocompleter for very specific case (such as your “enter, move and leave” example).

What do you think?

[mrkmg]

I love the idea of creating several classes to represent the different types of auto-completion. We could also include a Shortcut class to handle ex3.

I think Word and Shortcut will very simple and straight forward. I will start work on these soon.

The tough one will be something to handle ex2. There could be multiple ways of determining what the possible words could be, and how those words are determined will vary greatly from application to application. Here is a bullet point list of things we should consider when working on command-line

• Handle static lists of words for each position
• Handle dynamic lists of words, based on both position and context (currently typed in line)
• Handle unlimited amounts of params. For example: kill pid1 pid2 pid3 ... and so on. We should allow for autocomplete of these cases as well.
• Switch between autocomplete lists in the command. If i type kill and then hit tab, it has to know to use the pid list.

What about combining these? What if I want to get crazy and have shortcuts on my command line auto-complete. For example shortcut both the first and second params.

Also, should we implement a more versatile class, that allows for complete developer control? Kind of like how I have it now? We could call it generic. Or would it be easier just to allow then to pass a function exactly like it is now?

For the command line, I think we could have a custom, but simple syntax. It could use two arrays. One would be a command list, and the other a lookup table. In the command list, you can create a list of commands and their parameters. We can use familiar syntax to define dynamic parameters. A word alone matches on that word only. A $ means use the lookup table. A + at the end of a word means you can have as many of these as you want. We could do a +5 to indicate you need 5. Here is an example of how someone could setup the Autocomplete\CommandLine class.

$autocomplete = new Hoa\Console\Readline\Autocomplete\CommandLine([
    'move'=>'$person to $person_room',
    'enter'=>'$person+',
    'leave'=>'$person+',
    'do'=>'$action with $person'
],[
    'action'=>[ //returns array
        'attack',
        'dodge'
    ],
    'person'=>function($context, $word){ return People::search($word); }, //runs function
    'person_room'=>function($context, $word){
        $context = explode(" ",$context); // Break context into array of words
        $person = $context[count($context)-2]; //Get person in context, since we known it will be 2 positions behind
        return Rooms::possibleForPerson($person); //Return list of rooms for that person
    }
]);

This example allows autocomplete as follows: | indicates cursor position

> enter K<tab>
> enter Kevin |

> enter Kevin H<tab>
> enter Kevin Hoa |

> mo <tab>
> move |

> move Kev<tab>
> move Kevin |

> move Kevin to L<tab>
LivingRoom LightDeck
> move Kevin to Li|

> do a<tab>
> do attack |

> do attack with H<tab>
> do attack with Hoa |

I think this is a solid starting place to work from. Do you like the idea of a separate command and lookup array? I feel it is powerful and simple enough to deal with most situations easily. I can't think of any non-new-syntax way of accommodating even the most simple of command-line autocomplete situations.

Thoughts? Is my general process valid? Do you think its overly complicated?

[mrkmg]

Also, I think the idea of having the auto-complete functions being classes is super powerful as well. You can assign them to variables and switch them around as needed. This will allow a large amount of flexibility.

[mrkmg]

Examples of use for the previous Commit:

Word

$auto = new Hoa\Console\Readline\Autocomplete\Word([
    'bar',
    'boo',
    'baz',
    'foo'
]);
$reader = new Hoa\Console\Readline\Readline;
$reader->setAutocomplete($auto);
while(true){
    $reader->readLine('> ');
}

Shortcut

$auto = new Hoa\Console\Readline\Autocomplete\Shortcut([
    'ba'=>'bar',
    'bo'=>'boo',
    'bz'=>'baz',
    'f'=>'foo'
]);
$reader = new Hoa\Console\Readline\Readline;
$reader->setAutocomplete($auto);
while(true){
    $reader->readLine('> ');
}

CommandLine

$auto = new Hoa\Console\Readline\Autocomplete\CommandLine([
    'move'=>'$person to $person_room',
    'enter'=>'$person+',
    'leave'=>'$person+',
    'do'=>'$action with $person'
],[
    'action'=>[ //returns array
        'attack',
        'dodge'
    ],
    'person'=>function($context, $word){ echo 'ddd'; return getAllPeople(); }, //runs function
    'person_room'=>function($context, $word){
        $context = explode(" ",$context); // Break context into array of words
        $person = $context[count($context)-2]; //Get person in context, since we known it will be 2 positions behind
        return possibleForPerson($person); //Return list of rooms for that person
    }
]);

$reader = new Hoa\Console\Readline\Readline;
$reader->setAutocomplete($auto);
while(true){
    $reader->readLine('> ');
}

function getAllPeople(){
    return array(
        'Kevin',
        'Hoa'
    );
}

function possibleForPerson($person){
    if($person == 'Kevin') return array(
        'LivingRoom',
        'Kitchen',
        'LightRoom'
    );
    else return array(
        'Kitchen',
        'Basement'
    );
}

[hoaproject]

s/Autocomplete/Autocompleter/ in class names.

I think shortcut and word are pretty much the same, except that shortcut deletes word at the left of the cursor and adds at the right, while word only adds at the right. In this case, shortcut (or alias) would either extend word by adding ^W at the begining of each word, or be merge with word by allowing ^W (or \C-W) in a word. Thus:

$autocompleter = new Hoa\Console\Readline\Autocompleter\Word([
    'foo', // f<tab> => foo
    'bar', // b<tab> => bar
    'qux', // q<tab> => qux
    'bz^Wbaz' // bz<tab> => baz
])

But I have to admit I'm not sure of the pertinence of shortcut. It's a bastard autocompleter of word and a clever autocompleter that understands mistakes, do you see? And I'm not sure if it useful daily. I'm afraid you need to convince me :-/.

Finally, your CommandLine approach is very nice. I like it, but it does not fit to CommandLine. I would rather call it Placeholder or something like that. By saying command-line, I had imagined cmd --option=value -f -l -a -g input, do you see? Maybe am I wrong? Moreover, I see + in $person+. Do you want to introduce some repetition operators from regular expressions, such as: ?, +, * or {x,y}? If you do, then… it would be an interesting work, but we must well-define things.

I would recommend to add the “mistakes” feature in word. I can do this if it's too difficult for you. You have the Hoa\String\Search::approximated method that could help you, but I think we need another one, let's see.

Oh, I almost forget it. You talked about “combining” autocompleters. I think the best way would be by defining a chain: word, placeholder etc. If word does not find any result, then, we ask placeholder etc., do you see? I don't know if it is a good idea, but it is an idea.

Thank you for your time :-).

[mrkmg]

Completely agree with Autocompleter

First, I agree with CommandLine. I will rename that, but do you think PlaceHolder is a good name? Although it is not exactly like it, I was thinking of the Cisco IOS CLI interface. I'll think about it a little more, but in the mean time just rename it to be PlaceHolder

I was not really thinking of going full blown regex, just one operator to handle the idea of an infinite series of params. For example, if you have a script that controls lights in a bunch of rooms, you may want something like > on room1 room2 and > on room1 and >on room1 room3 room6. I don't know if the PlaceHolder would really benefit from a more complex regex-like system. Most cases can be implemented how it is now.

As for Shortcut, I do think it has useful applications, although I think Alias is a far better name. A simple example could be initials of names to expand out that name > kg<tab> to > Kevin Gravier, or we could have building numbers expand to street addresses > b100<tab> to > 123 parkway town st 01234. I like the idea of keeping Word and Alias separate though. Keeping the classes simple and specific while still allowing developer to roll his own I feel is the best way to go. If you want to do something crazy, then roll your own autocompleter.

I also dig the idea of multiple autocompleters being set at one. You could call setAutocomplete with multiple params. Each one would be checked for results in the order they were set. This could allow for easily inserting or removing items from possible autocompleters.

As for CommandLine, with the classic sense of argument switches. I have an idea for this as well. It could be called with a very similar structure as PlaceHolder.

$auto = new Hoa\Console\Readline\Autocompleter\CommandLine([
    'cmd'=>[
        'option'=>[
            'value'=>['bar','foo']
        ],
        'flag'=>[
            'short'=>'f'
        ],
        'input'=>[
            'short'=>'i',
            'value'=>function($context,$word){
                return ['baz','foz'];
            }
        ]
    ],
    'reset'=>[
        'all'=>[
            'short'=>'a',
        ],
        'name'=>[
            'short'=>'n',
            'value'=>['sue','joe']
        ]
    ]
]);

> cm<tab>
> cmd |

> cmd --o<tab>
> cmd --option |

> cmd  --option bar -i b<tab>
> cmd  --option bar -i baz |

> reset --name j<tab>
> reset --name joe |

What do you think of this? I think it is a solid place to start.

[mrkmg]

Example for CommandLine

$auto = new Hoa\Console\Readline\Autocompleter\CommandLine([
    'cmd'=>[
        'option'=>[
            'value'=>['bar','foo']
        ],
        'flag'=>[
            'short'=>'f'
        ],
        'input'=>[
            'short'=>'i',
            'value'=>function($context,$word){
                return ['baz','foz'];
            }
        ]
    ],
    'reset'=>[
        'all'=>[
            'short'=>'a',
        ],
        'name'=>[
            'short'=>'n',
            'value'=>['sue','joe']
        ]
    ]
]);
$reader = new Hoa\Console\Readline\Readline;
$reader->setAutocomplete($auto);
while(true){
    $reader->readLine('> ');
}

[hoaproject]

Ok, so let focus on the first autocompleter (I think we will need multiple issues in Github and a meta one, like this one).

Here are the two first steps:

1. Hoa\Console\Readline::_bindTab, it must be clean and allow autocompletion inside and at the end of word (abcdef<tab> or abc<tab>def);
2. make Hoa\Console\Readline\Autocompleter\Word

Next, we will make an aggregator of autocompleters (like for iterators, this is the same principle, we iterate over solutions and stop when one matches). To test the aggregator, I propose to make Hoa\Console\Readline\Autocompleter\Alias.

Then, we will continue with placeholder (I am aware that the name does not fit well, and I have understood the reference to Cisco library) and commandline. What do you think?

[mrkmg]

I won't be able to work on anything for this for a few days. After the holidays, I will start re-factoring _bindTab. Could you give me a list of things we should aim for with _bindTab? Starting points:

1. Autocomplete at anypoint in the string
   • Must be able to detect cursor position
2. Must be smart enough to figure out context, and properly pass that context to the autocompleter
3. Should be able to try a series of autocompleters, and use the first that returns results

Are there any other major points that I should focus on?

[hoaproject]

Hi,

I have written the _bindTab method and also the Hoa\Console\Readline\Autocompleter interface. Please, see commits bf426fb8ac6edd64cd908fd7e89035d298a539d4, 63120e699a954bc04a47c18304b3930ace766e57 and 720d77ce4b5ea317b33c32e39c5ae2660db8f343. It's a bit different from yours but the spirit is the same, I have just simplified the process to only find the prefix, call the autocompleter and deal with solutions.

Let me detail the _bindTab method for you:

• line 876: if no autocompleter is set, then we continue, and by default, the \t character is printed;
• line 879 and 880: get current position and the full line;
• line 883: if we are on the first column/character, we return $state which is equal to static::STATE_CONTINUE | static::STATE_NO_ECHO, which means “we continue to read the line and we do not echo/print the current typed character”;
• line 886 to 897: we find the current full-word (for example, if we do: abc<tab>def, the full word is abcdef);
• line 899: if the current full-word is a word-separator (\b), then we return (for example, if we do: abc <tab>def);
• line 902 and 903: we ask to autocompleter to complete the prefix (for example, if we do: abc<tab>def, the prefix is abc);
• line 905: if no solution, then return;
• line 908: if many soutions, print a newline (line 911), print all solutions (line 912), print the line (line 913) and replace the cursor (line 913 to 916);
• line 922 to 927: we have one solution, so we insert it: print the solution, clean the right-side of the line, print the tail and replace the cursor, and we finally use the insertLine method to update line properties, such as _line, _lineLength and _lineCurrent;
• line 929: we return!

This was the easy part. Autocompleter needs one method by now: complete ( $prefix ). Do you think we need more data?

I have tried to make a menu à la Zsh, but it was not so easy ;-). For a next update I think.

I will add you to the credits later (when it will be finished, it's simpler). You are welcome on IRC (freenode, #hoaproject) to discuss about the following patches :-).

[hoaproject]

I have added the Hoa\Console\Cursor class (please, see a95f67e18979fd3019ecb930588f5ef48f64717a). It simplifies a lot the code readibility of Hoa\Console\Readline & co. (please, see d65a3eff741a5a99a23fe55bfe00f4741deac2fa).

A good example is lines 905 to 909: we save the cursor position, then we move down of one line and we go the extreme left column, then we clear below, we print the solution and we restore the cursor position. Consequently, when we auto-complete a word, we do not print a new prompt each time we have many solutions but we keep the same prompt and things happen below:

> foo<tab>
bar baz qux

instead of:

> foo<tab>
bar baz qux
> foo

[hoaproject]

@mrkmg ping?

[Hywan]

Hi,

Last commits (please, see abe1e8802ae858a096da284a670e990ef7eee200, 2f9b71eec76a4e216b59d78fed624f32bac28902, 7d02a90df12d081c8faee6743ac664bb9ab107b2, e421ebba34e483da9e3ec694ae87b247f58559c2 and cb81f4f2b8b8ac7c95d4580f0f9af39b03d979e7) introduce a menu to select one solution among many from the auto-completer (à la Zsh). Yes, it's very cool :-). or temporary video. We are able to navigate in the menu with keyboard arrows (very useful).

Also, we have the word autocompleter (it could be ameliorated).

Well, it starts to be very interesting isn't it?

[mrkmg]

Wow that's amazing. I think this is going beyond my abilities. I would still love to help, but I feel I would need some guidance.

[hoaproject]

You're fully able to write other auto-completers as you have already proposed! I have just focused on the presentation of solutions given by auto-completers. I encourage you to continue :-).

[mrkmg]

I did not want to close that. That was an accident. But now that it is closed I will make seperate pull requests for each autocompleter

[hoaproject]

Ok, excellent!