Make edit:command-history easier, and faster, to use with tools like fzf

[krader1961]

See #1051 for what prompted me to open this issue.

At least one common use-case of the edit:command-history command would be easier and faster (i.e., more efficient) with some enhancements to that command:

1) An option to output the commands in newest-to-oldest order (keeping the default oldest-to-newest).

1) An option to eliminate duplicate commands regardless of the output order. "Duplicate" here means comparing just the "cmd" member of the default map output.

1) An option to request just the "cmd" string be emitted rather than a map of all command attributes.

1) Maybe: An option to limit the output to the N newest commands; without regard to the order of the output. Note that the take command only works as a substitute for this option if the option to output the commands in newest-to-oldest order is used. On the other hand this option may not make much sense when coupled with oldest-to-newest order (the default). Regardless of whether or not it is treated as the first or last N commands. Thus the "maybe" prefix since it may be better to omit this option.

Using the first three options makes it easy to do command selection using the popular fzf utility. It should also improve the time to produce a fzf menu by at least an order of magnitude.

The duplicate command elimination option might warrant more discussion. Specifically, whether elvish should always do duplicate elimination when updating the command data store. In all my years using ksh and zsh as my day-to-day interactive shells I can't ever recall a situation where retaining each individual use of a command in the history was useful. As opposed to being an annoyance when I listed the N most recent commands I ran or was stepping backward through my command history; e.g., via [up-arrow]. Note that fish always does duplicate elimination. I can't recall ever hearing anyone complain about the deduplication.

[krader1961]

See also #568 which asked for a way to access a deduplicated command history more than two years ago.

[krader1961]

A recent question on IRC reminded me that I, like @zzamboni, had added some functions to my rc.elv several months ago to use the fzf program rather than the builtin histlist mode but hadn't bound those functions to Ctrl-R. So I did so and was happy with fzf but not how long it took before the fzf query prompt was usable.

On my primary MacPro server edit:command-history >/dev/null completes in 55 ms. The following function takes 760 ms (~13.8 times longer):

edit:command-history | each [hist-entry]{
    print $hist-entry[cmd]"\000"
} | perl -n0e 'print unless $h{$_}++' >/dev/null

A pure Elvish solution takes 2849 ms (~3.7 and ~51.8 times longer than the perl and unmodified history solutions respectively):

seen = [&]
edit:command-history |
    each [hist-entry]{
        cmd = $hist-entry[cmd]
        if (has-key $seen $cmd) {
        continue
        }
        seen[$cmd] = $true
        print $cmd"\000"
    } >/dev/null

Implementing a couple of the proposed enhancements should dramatically improve the experience of using fzf for selecting history entries. Especially if combined with a to-lines enhancement to control the EOL character.

[krader1961]

I have a proof-of-concept change that adds a &dedup and &newest-first option; thus addressing items one and two in my original problem statement. It dramatically reduces the cost of feeding such output to an external command. On my system the following takes 151.819 ms while the solution using perl requires 728.985 ms -- a factor of 4.8 times faster. So much faster that pressing Ctrl-R no longer has any visible artifacts before I can interact with fzf.

> edit:command-history &dedup &newest-first |
    each [hist-entry]{ print $hist-entry[cmd]"\000" } > /dev/null

It's certain to be even faster if we can replace the each... print... in the above pipeline with to-lines &eol="\000" (or an equivalent capability) and edit:command-history gains a &cmd-only option to output just the command text rather than a command map.

P.S., Note that outputting all commands takes 511.519 ms on my system. Which is 3.4 times slower than outputting just the non-duplicates:

> time { edit:command-history | each [hist-entry]{ print $hist-entry[cmd]"\000" } > /dev/null }

Which, again, leads to the question of whether Elvish should even be recording each non-unique instance of a command since doing so isn't particularly useful and bloats the command history for little benefit.

[krader1961]

On my system the following takes 151.819 ms while the solution using perl requires 728.985 ms -- a factor of 4.8 times faster.

Also, it's important to note the solution using perl is wrong as it does not correctly preserve the order of the commands. Whereas the &dedup option correctly preserves the order by retaining the most recent, rather than the oldest, instance of a command.

[krader1961]

Implementing to-lines &null, along with the other enhancements discussed above, reduces the cost of constructing the input for the fzf program from 760 ms (for the perl based, incorrect, solution) to 34.7 ms. On my system pressing Ctrl-R now has no visible lag before I can search my command history.

[krader1961]

Using both PR #1307 and PR #1308 allows a very efficient Ctrl-R binding that uses the fzf program for selection. So efficient it is indistinguishable from the builtin histlist navigation mode with respect to initialization time:

# Filter the command history through the fzf program. This is normally bound
# to Ctrl-R but can be invoked explicitly by running `history`.
fn history []{
  # If the user presses Escape to cancel the fzf operation it will exit with a
  # non-zero status. We want to ignore that exception; hence the status
  # capture. We could use an explicit `try ...except...` but this is simpler.
  _ = ?(edit:current-command = (
    edit:command-history &dedup &newest-first &cmd-only | to-lines &null |
    fzf --no-sort --read0 --layout=reverse --info=hidden --exact ^
      --query=$edit:current-command)
  )
}

edit:insert:binding[Ctrl-R] = []{ history >/dev/tty 2>&1 }

[krader1961]

For anyone who stumbles upon this issue in the future.... When writing the necessary code and responding to review feedback some of the details of this proposal were changed. This is the code I now have in my ~/.elvish/rc.elv init script:

# Filter the command history through the fzf program. This is normally bound
# to Ctrl-R.
fn history []{
  var new-cmd = (
    edit:command-history &dedup &newest-first &cmd-only |
    to-terminated "\x00" |
    try {
      fzf --no-sort --read0 --layout=reverse --info=hidden --exact ^
        --query=$edit:current-command
    } except {
      # If the user presses [Escape] to cancel the fzf operation it will exit
      # with a non-zero status. Ignore that we ran this function in that case.
      return
    }
  )
  edit:current-command = $new-cmd
}

edit:insert:binding[Ctrl-R] = []{ history >/dev/tty 2>&1 }

[krader1961]

Everything needed to make binding Ctrl-R to fzf is in place for the upcoming 0.16.0 release so this is resolved.

[aca]

for 0.17, slight improvement.

• handle multiline commands
• fallback
• ..

  fn fzf_history {||
  if ( not (has-external "fzf") ) {
  edit:history:start
  return
  }
  var new-cmd = (
  edit:command-history &dedup &newest-first &cmd-only |
  to-terminated "\x00" |
  try {
    fzf --no-multi --height=30% --no-sort --read0 --info=hidden --exact --query=$edit:current-command | slurp
  } except {
    edit:redraw &full=$true
    return
  }
  )
  edit:redraw &full=$true
  set edit:current-command = $new-cmd
  }
  set edit:insert:binding[Ctrl-R] = {|| fzf_history >/dev/tty 2>&1 }

[ezh]

Note. @aca Solution is great, but cursor jumps to the next line. Fix set edit:current-command = (str:trim-space $new-cmd)

And for those who like to select history, simply press enter:

fn is_readline_empty { 
  # Readline buffer contains only whitespace.
  re:match '^\s*$' $edit:current-command 
} 

set edit:insert:binding[Enter] = { 
  if (is_readline_empty) { 
    # If I hit Enter with an empty readline, it launches fzf with command history
    set edit:current-command = ' history'
    edit:smart-enter 
    # But you can do other things, e.g. ignore the keypress, or delete the unneeded whitespace from readline buffer
  } else {
    # If readline buffer contains non-whitespace character, accept the command. 
    edit:smart-enter 
  } 
} 

[RyanGreenup]

For anyone who comes across this, note that except is now catch so @aca 's solution needs to be:

11c11
<     } except {
---
>     } catch {