history should be customizable by the end user (HN thread summary)

[andychu]

This is a summary of Better Bash History discussion on HN

@emdash There is lots of interesting feedback here about history, including someone putting their history in an sqlite DB! Bash is crufty yet it's pretty amazing what people manage to hack on to it!

Some takeaways:

• Implement $PROMPT_COMMAND because lot of people are using it!
• Run fzf, lots of people are using it and like it.

Things people are doing:

• per-directory history (how do you do this?) -- I set up my bash config to store history separately for each working directory (ie separate history files that I store under a $HOME/.bash_history_d directory) – so as I cd into a project directory, the history is populated with the commands that I've run within that directory.
  • https://github.com/YenTheFirst/dotfiles/blob/master/.bash_cd_hook.sh

. .bash_cd_hook.sh
export PROMPT_COMMAND="history -a; check_cd; ${PROMPT_COMMAND:-:}"

• I integrated all of these history options in my project Sensible Bash [1], an attempt at saner Bash defaults
  • https://github.com/mrzool/bash-sensible
• I made an ebook [2] out of a series which appeared on this blog, titled “Unix as IDE”.
  • https://github.com/mrzool/unix-as-ide
• Because this comes up every so often, I'll post this link again: I wrote a little widget that stores your bash history in a sqlite db.
  • https://github.com/thenewwazoo/bash-history-sqlite
• https://eli.thegreenplace.net/2016/persistent-history-in-bash-redux/
  • Before doing this, the only way I had to remember which commands/flags are needed to run something was to write it down in all kinds of notes files, personal wikis and so on. It was cumbersome, unorganized and time-consuming to reuse. With the .persistent_history file automatically populated by Bash from any terminal I'm typing into, and being kept in a Git repository for safekeeping, I have quick access to any command I ever ran.
• FZF makes something you do hundreds of times a day (history and file opening on zsh, bash, fish, whatever) indisputably better.
• I use zsh, and I have a thing in my profile ... that will rewrite the command using the absolute path of the directory I cd'd into. Thus, all of the cd commands that get written to my history use absolute paths and are re-usable no matter where I am
  • It would be cool if this sort of functionality were generalizable to all commands where you type a relative path, not just cd.*
  • https://github.com/dasl-/settings/blob/3143bbfe23bd75c3e35c7...
  • note: osh could give you an argv array as well as the raw text?

History:

• Bash's broken history is what finally got me to switch to Zsh.

Other interesting quotes, not about history:

• autocompletion in fish takes into account the directory where you are at, so it will autocomplete commands ran in the same directory before other commands.
• I do something similar, but instead of current directory I have a notion of context that is carried by an environment variable set in a GNU screen process (and therefore inherited by its children) which drives logic in my .bashrc -- this is similar to the way I work

[emdash]

Good resources, I will look at FZF. I had not heard of it.

[andychu]

OK thanks!

The thread helped clarify what I'd want in Oil for history:

https://news.ycombinator.com/item?id=20057323

Basically it should be a structured text file. Coincidentally I wrote this the other day because @drwilly is working on xargs, which also needs structured data:

https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

The point is that you probably want history in SQL for two reasons:

• Because you want to query history without parsing text (usually done incorrectly). That can be done with the simple TSV2 proposal. Oil will parse that so you don't have to, but you can also grep the file if you want.
• For concurrency. I don't think this is a problem for append only files (although admittedly I wonder how HISTCONTROL=erasedups is implemented).

I think what'ss weird about bash history is this:

• history is kept in a buffer in memory in each instance
• on exit, the whole thing is appended to a file.

That's why it's so weird between "sessions"! At least I think so.

This thread also points to the benefit of having a real programming language in shell. Instead of a PROMPT_COMMAND string to evaluate, we could have a hook:

func whenCommandFinished(command_string, evaluated_argv, timestamp, status, elapsed_time, ...) {
  # do something yourself with history, like put it in a sqlite database, or upload it to your own server, etc.
}

[emdash]

Pretty much agree.

The other thing I would add is making history more robust in the presence of events which kill the shell before it has a chance to save its history.

Either a function hook or a trap.

At least one comment hints at how controversial this sort of thing will be :P

On Fri, May 31, 2019, 10:03 andychu notifications@github.com wrote:

OK thanks!

The thread helped clarify what I'd want in Oil for history:

https://news.ycombinator.com/item?id=20057323

And coincidentally I wrote this the other day because @drwilly https://github.com/drwilly is working on xargs, which also needs structured data:

https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

Anyway, the point is that you probably want history in SQL for two reasons:

• because you want fields without parsing them (usually done incorrectly). That can be done with the simple TSV2 proposal. Oil will parse that so you don't have to, but you can also grep the file if you want.
• for concurrency. I don't think this is a problem for append only files (although admittedly I wonder how HISTCONTROL=erasedups is implemented).

I think what is weird about bash is this:

• history is kept in a buffer in memory
• on shell's exit, the whole thing is appended to a file.

That's why it's so weird between sessions! At least I think so.

This thread also points to the benefit of having a real programming language in shell. Instead of a PROMPT_COMMAND string to evaluate, we could have a hook:

func whenCommandFinished(command_string, evaluated_argv, timestamp, status, elapsed_time, ...) {

do something yourself with history, like put it in a sqlite database, or upload it to your own server, etc.

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oilshell/oil/issues/320?email_source=notifications&email_token=AAAC3W47UY3GL4SQM7P7LILPYFK4XA5CNFSM4HRPAND2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWVZZLA#issuecomment-497786028, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAC3WYTRSOL56T32ZVLJTLPYFK4XANCNFSM4HRPANDQ .

[curusarn]

I'm designing a history extension/replacement for bash and zsh and instead of having awkward two-level history I have a daemon-client type situation. Maybe oilshell could do something similar.

[andychu]

@curusarn I'd be interested in seeing what you've done for bash and zsh!

It would be interesting for Oil to support some kind of external protocol so people can plug in their own logic.

Although one limitation is that we use readline for history, which has its own file format. But bash has that issue too, so I'm curious how you solved it.

[curusarn]

@andychu Sorry for the long silence.

Readline is pretty difficult to work with.

The idea behind my project is to provide context-based history enhancement for zsh and bash. So I have a daemon that records the history with metadata and then handles all the interaction with history.

There are multiple ways I need to integrate with the shell:

• Collecting the commands (I'm recording the history ad-hoc because I want extra data e.g.: directory, git, exit status, ...)
• Arrow key bindings (Pressing up/down to access history, prefix search, etc ...)
• Ctrl+R binding

To collect the history I run custom functions before/after each command using bash-preexec. It uses $PROMPT_COMMAND and DEBUG trap. It works with some minor issues.

Executing custom commands through readline bindings is problematic. There is bind -x builtin in bash to bind commands to key sequences but readline redraws the whole prompt when the command executes which is pretty slow. I asked in a readline mailing list about it but they told me it's not an easy thing to change at all (unsurprisingly). The redraw itself is so slow I have disabled arrow key bindings in bash in my project for now.

I didn't implement the binding for Ctrl+R yet but I want it to open an app similar to hstr except my app will use all the extra data for searching. There should be no problem to do this since hstr already works in bash.

All of the above is much easier to do in zsh then in bash.
Integrating with bash and readline is always a painful experience riddled with weird behavior and missing features. Integrating with zsh and ZLE (zsh line editor) is fairly straight forward.

Link to my project for more context and more info.

I'm not sure exactly how much of the difficulties I encountered were caused by readline itself and how much I can blame bash. My intuition is that the biggest problem are the bindings because anything else than native readline history functions takes the performance hit when redrawing.

[abathur]

I've also been picking at a bash-specific history module (opinionated; uses sqlite, also stays compatible with the underlying history functionality). I'm not promoting it for others to use, yet, but it's been in my shell for close to a year now, I guess. Don't have time for a longer comment atm, but I'll get notifications, and you'll know I have opinions. :)

[andychu]

@abathur What interface are you using to glue bash together with sqlite? Is it $PROMPT_COMMAND ?

That seems like a fragile interface but I suppose it's already there and a lot of people are using it.

I'm more interested in providing a good interface for Oil than developing this myself. Though it already implements $PROMPT_COMMAND ... maybe that's good enough but I'd be surprised if we couldn't do better.

[abathur]

Yep.

Indirectly, though--I've abstracted most messy implementation parts into a shell library (to keep them away from the history module itself, minimize duplicated work in other parts of my profile, and make it possible to share the PROMPT_COMMAND).

I do think Oil can do better, though I'm not sure it's incompatible with supporting PROMPT_COMMAND. Can probably discuss on zulip sometime soon.

[curusarn]

Oil could support preexec and precmd hooks that are supported by zsh. These could be easily used to plug in arbitrary history extensions to oil shell.

precmd Executed before each prompt. Note that precommand functions are not re-executed simply because the command line is redrawn, as happens, for example, when a notification about an exiting job is displayed.

preexec Executed just after a command has been read and is about to be executed. If the history mechanism is active (regardless of whether the line was discarded from the history buffer), the string that the user typed is passed as the first argument, otherwise it is an empty string. The actual command that will be executed (including expanded aliases) is passed in two different forms: the second argument is a single-line, size-limited version of the command (with things like function bodies elided); the third argument contains the full text that is being executed.

[andychu]

Yeah I think those hooks are a good idea, and the different versions of the command are what I wanted to sort out:

• the literal text the user typed, e.g. echo $FOO
• the expanded argv, e.g. ['echo', 'foovalue']. I would want to keep this as an argv array, not "text" as my reading of the zsh manual excerpt implies.

pre- and post- alias expansion is another possibility, and probably not hard, although I'm not sure how useful it is. (Almost all aliases can simply be shell functions.)

This distinction appears to be missing from bash's history mechanism on the "output side".

I would also favor something like prompt_hook() { } over $PROMPT_COMMAND.

Just like bash has command_not_found() { } now.

This also relates to $PS1 and $PS4. Basically the string hooks are annoying and error prone IMO.

[andychu]

related:

• better $PS1 is #498
• better $PS4 is #583

[andychu]

https://news.ycombinator.com/item?id=25946055

Good comment here

My histories are always saved because each shell instance gets its own HISTFILE, like so:

  export HISTFILE=$HOME/.history/${TTY##\*/}.$SHLVL

As I use different terminal windows for different tasks, this keeps history files rather concise thematically.

And I let the shell add timestamps too, so I can grep for entries produced during a certain time span:

zsh:

  setopt EXTENDEDHISTORY # add timestamps

bash:

  HISTTIMEFORMAT="%F %T "