bash history timestamp interaction

[rux616]

ble version: 0.4.0-devel2+efe1e81 Bash version: 5.0.16(1)-release,x86_64-pc-linux-gnu

((Before I begin, I just wanted to say that this project is amazing. Thank you for making it!))

It seems that ble.sh has some weird interactions with bash's history mechanism. For reference, here are my history-related settings in my bashrc:

# Set history file to be different than default to avoid accidental overwrites
# http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login
export HISTFILE="${XDG_DATA_HOME}/bash/history"

# Set timestamp format
export HISTTIMEFORMAT="[%F %T]  "

# Avoid duplicates
export HISTCONTROL=ignoredups:erasedups

# Unlimited history
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTSIZE=
export HISTFILESIZE=

# When the shell exits, append to the history file instead of overwriting it
shopt -s histappend

# After each command, append to the history file and reread it
# http://superuser.com/questions/20900/bash-history-loss
#export PROMPT_COMMAND="${PROMPT_COMMAND:+"${PROMPT_COMMAND}; "}history -a; history -c; history -r"

Given that I have HISTTIMEFORMAT set, I would expect that the history that gets appended to HISTFILE on a shell exit would have a '#\' string inserted on the line before each command is recorded. But that does not seem to be the case right now. When a ble-attached shell exits, it appears to just dump its entire command history directly into HISTFILE without timestamp lines.

Also, because bash is a bit dim about sharing history between different shells and the fact that crashes are a thing, I usually have the PROMPT_COMMAND modification active so it writes to history every time a command is executed, though I commented it out while working on tracking this issue.

[akinomyoga]

Thank you very much for the report!

((Before I begin, I just wanted to say that this project is amazing. Thank you for making it!))

Thank you for your encouraging words!

Given that I have HISTTIMEFORMAT set, I would expect that the history that gets appended to HISTFILE on a shell exit would have a '#' string

This is just because lines are appended to HISTFILE by ble.sh but not by Bash when ble.sh is activated, and I haven't implemented #<EPOCHTIME> in ble.sh. I am going to implement it.

Also, because bash is a bit dim about sharing history between different shells

ble.sh has a setting bleopt history_share=1 for this. An idiom history -a; history -c; history -r causes the reload of entire history which is not so efficient. In particular, ble.sh needs to re-initialize its history data every time history -r is called. bleopt history_share=1 uses a more natural way to implement the same feature. Also, the idiom caused a problem in ble.sh before at #24 though I have added a workaround for that.

[rux616]

This is just because lines are appended to HISTFILE by ble.sh but not by Bash when ble.sh is activated, and I haven't implemented #<EPOCHTIME> in ble.sh. I am going to implement it.

That's great to hear, I look forward to that.

ble.sh has a setting bleopt history_share=1 for this.

The setting is very useful and is already active in my init.sh. :)

An idiom history -a; history -c; history -r causes the reload of entire history which is not so efficient. In particular, ble.sh needs to re-initialize its history data every time history -r is called. bleopt history_share=1 uses a more natural way to implement the same feature. Also, the idiom caused a problem in ble.sh before at #24 though I have added a workaround for that.

Yeah, I've never liked the history -a/-c/-r pattern, but prior to ble.sh that was basically the only option if you wanted to stick with bash versus zsh or fish.

Will ble.sh work okay with using history -a in the PROMPT_COMMAND or will it cause lots of duplicated commands in HISTFILE when a ble-attached shell exits? I'm kind of imagining a solution that looks something like this: https://unix.stackexchange.com/a/556267

[akinomyoga]

This is just because lines are appended to HISTFILE by ble.sh but not by Bash when ble.sh is activated, and I haven't implemented # in ble.sh. I am going to implement it.

That's great to hear, I look forward to that.

I have supported 4bcbd71. Could you test if it works as you expected?

Will ble.sh work okay with using history -a in the PROMPT_COMMAND or will it cause lots of duplicated commands in HISTFILE when a ble-attached shell exits?

I thought there is no problem with specifying history -a in PROMPT_COMMAND, but maybe I misunderstand what you mean by "duplicated commands". Have you actually seen the duplicate lines in HISTFILE with this setting?

[rux616]

I have supported 4bcbd71. Could you test if it works as you expected?

It ... mostly works. (For the record, for these tests, I have disabled all history -* commands in PROMPT_COMMAND.)

Test methodology:

• connect to remote GCP instance without ble.sh active (session-1)
• use (session-1) to do any changes that need to be made, like alter settings in bash rcfile and blesh rcfile, also has an active tail -f $HISTFILE
• restore an existing backup HISTFILE with timestamps in it
• once changes are made, connect to remote GCP instance again with ble.sh active (session-2)
• in (session-2), issue commands
  • echo this
  • echo is
  • echo a
  • echo test
  • ls
  • ls -al
  • exit
• record what happens
• repeat from step 2

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	OFF
shopt -s histappend	OFF
bleopt history_share=1	OFF

Action	Result
Entering commands	-HISTFILE not updated immediately (expected)
Exiting ble-attached shell	-HISTFILE gets rewritten (expected) - session-local commands added at bottom (expected) - existing timestamps stripped (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	ON
shopt -s histappend	OFF
bleopt history_share=1	OFF

Action	Result
Entering commands	-HISTFILE not updated immediately (expected)
Exiting ble-attached shell	-HISTFILE gets rewritten (expected) - session-local commands added at bottom with timestamps (expected) - existing timestamps preserved (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	OFF
shopt -s histappend	ON
bleopt history_share=1	OFF

Action	Result
Entering commands	-HISTFILE not updated immediately (expected)
Exiting ble-attached shell	-HISTFILE gets appended to (expected) - session-local commands added at bottom without timestamps (expected) - existing timestamps reserved (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	OFF
shopt -s histappend	OFF
bleopt history_share=1	ON

Action	Result
Entering commands	- HISTFILE is updated immediately without timestamp (expected) - get error -bash: /run/user/1001/blesh/639344.history.read.part: No such file or directory on first command but seemingly only if HISTFILE has timestamps in it (error)
Exiting ble-attached shell	HISTFILE gets rewritten (expected), - existing timestamps stripped (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	ON
shopt -s histappend	ON
bleopt history_share=1	OFF

Action	Result
Entering commands	-HISTFILE not updated immediately (expected)
Exiting ble-attached shell	-HISTFILE gets appended to (expected) - session-local commands added at bottom with timestamps (expected) - existing timestamps preserved (expected) - timestamps could be out of order (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	ON
shopt -s histappend	OFF
bleopt history_share=1	ON

Action	Result
Entering commands	-HISTFILE updated immediately with timestamps (expected) - get error -bash: /run/user/1001/blesh/665445.history.read.part: No such file or directory on most commands (except for echo is for some reason) (error)
Exiting ble-attached shell	-HISTFILE gets appended to (expected) - existing timestamps preserved (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	OFF
shopt -s histappend	ON
bleopt history_share=1	ON

Action	Result
Entering commands	-HISTFILE updated immediately without timestamps (expected) - get error -bash: /run/user/1001/blesh/684848.history.read.part: No such file or directory on first command (error)
Exiting ble-attached shell	-HISTFILE gets appended to (expected) - existing timestamps preserved (expected)

Case:	Option	State
export HISTTIMEFORMAT="[%F %T] "	ON
shopt -s histappend	ON
bleopt history_share=1	ON

Action	Result
Entering commands	-HISTFILE updated immediately with timestamps (expected) - get error -bash: /run/user/1001/blesh/695343.history.read.part: No such file or directory on a few commands (echo this, echo test, ls, ls -al) (error)
Exiting ble-attached shell	-HISTFILE gets appended to (expected) - existing timestamps preserved (expected)

Will ble.sh work okay with using history -a in the PROMPT_COMMAND or will it cause lots of duplicated commands in HISTFILE when a ble-attached shell exits?

I thought there is no problem with specifying history -a in PROMPT_COMMAND, but maybe I misunderstand what you mean by "duplicated commands". Have you actually seen the duplicate lines in HISTFILE with this setting?

No, sorry. I was misunderstanding how bleopt history_share=1 works. You can disregard that question.

[akinomyoga]

Thank you for your detailed testing! This is great. This was a simple mistake 350bb15.

[akinomyoga]

I was misunderstanding how bleopt history_share=1 works.

OK. I guess the trick that confuses you is the fact that ble.sh overwrites the builtin history by a function. ble.sh tracks every call of history to synchronize its internal data with that of Bash.

$ type history
history is a function
history ()
{
    ble/builtin/history "$@"
}

So, if you have builtin history -a in PROMPT_COMMAND to bypass the ble.sh history synchronization, you will get duplicate lines in HISTFILE. You have a good insight!

[rux616]

Thank you for your detailed testing! This is great. This was a simple mistake 350bb15.

Good to hear it's a simple one. I can confirm it's fixed in 350bb15.

I was misunderstanding how bleopt history_share=1 works.

OK. I guess the trick that confuses you is the fact that ble.sh overwrites the builtin history by a function. ble.sh tracks every call of history to synchronize its internal data with that of Bash.

$ type history
history is a function
history ()
{
    ble/builtin/history "$@"
}

So, if you have builtin history -a in PROMPT_COMMAND to bypass the ble.sh history synchronization, you will get duplicate lines in HISTFILE. You have a good insight!

Exactly. So I've gated my PROMPT_COMMAND history -* modifications behind ((_ble_bash)) so they will still run and synchronize history, but only when ble.sh isn't loaded.

[akinomyoga]

OK. Thank you for your report and detailed testing!