dvorka / hstr

bash and zsh shell history suggest box - easily view, navigate, search and manage your command history.
http://me.mindforger.com
Apache License 2.0
4.08k stars 231 forks source link

hstr stops working on linux >=6.2.0 (depending on kernel config) #478

Closed leapfog closed 1 year ago

leapfog commented 1 year ago

Just wanted to leave a note, that hstr stops working with linux kernel 6.2.0, when CONFIG_LEGACY_TIOCSTI isn't set. So maybe you want to add that information to a trouble shooting guide or FAQ or something.

(re: "stops working": the history can still be browsed, but cannot be inserted anymore)

BlueMax commented 1 year ago

Edit: Never mind, probably only switchable with CONFIG_LEGACY_TIOCSTI=y.

How is this meant to be done?

This functionality can be changed at runtime with the dev.tty.legacy_tiocsti sysctl. This configuration option sets the default value of the sysctl. https://cateee.net/lkddb/web-lkddb/LEGACY_TIOCSTI.html

It can't be set though.

# sysctl -w dev.tty.legacy_tiocsti=1
sysctl: setting key "dev.tty.legacy_tiocsti": Invalid argument
# echo 1 > /proc/sys/dev/tty/legacy_tiocsti
bash: echo: write error: Invalid argument

Arch Linux 6.2.1

sxe commented 1 year ago

I came here as well cause hstr stopped working. Could we get an official reply on how this situation will be handled? Will hstr be adjusted to work without it? I would rather not enable a legacy option.

Cheers

rajakesar commented 1 year ago

❯ uname -r 6.2.1-arch1-1

❯ date Thu Mar 2 10:38:30 AM EST 2023

Hstr is confirmed NOT working on the above kernel as of the time above.

jakedane commented 1 year ago

If you run zgrep CONFIG_LEGACY_TIOCSTI /proc/config.gz you can see CONFIG_LEGACY_TIOCSTI is not set for 6.2.1-arch1-1. If indeed hstr insertion is not working because CONFIG_LEGACY_TIOCSTI is not set then this is why for 6.2.1-arch1-1.

mmeier86 commented 1 year ago

I can confirm that it's the CONFIG_LEGACY_TIOCSTI option. I had the same problem after I updated my Gentoo to 6.2, and after reenabling that option in my Kernel, it started to work again with 6.2.

leapfog commented 1 year ago

Seems fzf's history search is still working, even without CONFIG_LEGACY_TIOCSTI, so there is a way.

jakedane commented 1 year ago

Seems fzf's history search is still working, even without CONFIG_LEGACY_TIOCSTI, so there is a way.

fzf has a bash function around it for the history function, that sets the readline input buffer with the history line that was selected in fzf. The same can be done with hstr!

What I did:

  1. Build hstr with DEBUG_NO_TIOCSTI set. You just need to uncomment this line and recompile hstr: https://github.com/dvorka/hstr/blob/master/src/hstr_utils.c#L29. With DEBUG_NO_TIOCSTI set hstr doesn't use the insecure TIOCSTI and instead prints the selected history line to stderr. On Arch Linux I did this with:
    auracle download hstr
    cd hstr
    makepkg -so
    sed -i -r 's|//(#define DEBUG_NO_TIOCSTI)|\1|' src/hstr-*/src/hstr_utils.c
    makepkg -e
    sudo pacman -U hstr-*.zst
  2. We don't want the selected history line on stderr, we want it in the readline input buffer. I couldn't fully follow how fzf was doing that but on the wiki there is an example for bash: https://github.com/junegunn/fzf/wiki/Examples#with-write-to-terminal-capabilities. I used the 3rd example, the one starting with # re-wrote the script above, and adjusted it a bit for hstr. Basically: call hstr, redirect its stderr to a file so that can be read back into a variable and then pass that variable to the function from the wiki that sets the readline input buffer. Add this to your ~/.bashrc:
    
    bind '"\C-r": "\C-x1\e^\er"'
    bind -x '"\C-x1": __hstr';

__hstr () { hstr 2> ~/.hstr.tmp hstr_tmp=$(< ~/.hstr.tmp) __ehc "$hstr_tmp" }

__ehc() { if [[ -n $1 ]] then bind '"\er": redraw-current-line' bind '"\e^": magic-space' READLINE_LINE=${READLINE_LINE:+${READLINE_LINE:0:READLINE_POINT}}${1}${READLINE_LINE:+${READLINE_LINE:READLINE_POINT}} READLINE_POINT=$(( READLINE_POINT + ${#1} )) else bind '"\er":' bind '"\e^":' fi }

3. And you need to **disable** the default hstr keybindings in your `~/.bashrc` so comment out these lines:

if this is interactive shell, then bind hstr to Ctrl-r (for Vi mode check doc)

if [[ $- =~ .i. ]]; then bind '"\C-r": "\C-a hstr -- \C-j"'; fi

if this is interactive shell, then bind 'kill last command' to Ctrl-x k

if [[ $- =~ .i. ]]; then bind '"\C-xk": "\C-a hstr -k \C-j"'; fi


Now if I open a new terminal, press Ctrl+R, hstr works again and the selected history line is placed on the readline input.

I don't fully understand all of the above — and it can probably be cleaned up and made nicer — but at least I have a working hstr again.
BlueMax commented 1 year ago

CopyQ is another candidate that still works. I'm not sure but i think they load the clipboard and send a keystroke via X11 to paste the text.

papavlos commented 1 year ago

Thanks @jakedane for the workaround. It works! There is a small issue - pressing Enter on selected command in history places the command into input buffer but without automatic execution. I have to press Enter again, but this is very small issue.

Mte90 commented 1 year ago

I am on debian sid and stopped working too with 6.2.2-2-siduction-amd64 #1 SMP PREEMPT_DYNAMIC siduction 6.2-2.1 (2023-03-06) x86_64 GNU/Linux

apt source hstr
sed -i -r 's|//(#define DEBUG_NO_TIOCSTI)|\1|' src/hstr_utils.c
debuild -uc -us

Compiled with the patch, it still not work. I just get the command in the terminal after the selection compared to before.

immagine

jakedane commented 1 year ago

Compiled with the patch, it still not work. I just get the command in the terminal after the selection compared to before.

That sounds like you did step 1 from https://github.com/dvorka/hstr/issues/478#issuecomment-1454727507 but didn't proceed with step 2 and 3 to modify your .bashrc file (and open a new terminal after).

Mte90 commented 1 year ago

So I tried that bash changes and in this way works only Ctrl+r not hh as example (I tried changing the bash-completion script):

#
# Copyright (C) 2014-2022  Martin Dvorak <martin.dvorak@mindforger.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Bash completion support for HSTR

# Source this file or install it to /usr/share/bash-completion/completions or /etc/bash_completion.d/
# See https://iridakos.com/tutorials/2018/03/01/bash-programmable-completion-tutorial.html
# help complete
# complete -W "--favorites --kill-last-command --non-interactive --show-configuration --show-zsd-configuration --show-blacklist --version --help" hstr

bind '"\C-r": "\C-x1\e^\er"'
bind -x '"\C-x1": _hstr';

#_hstr()
#{
#        local cur prev OPTS
#        COMPREPLY=()
#        cur="${COMP_WORDS[COMP_CWORD]}"
#        prev="${COMP_WORDS[COMP_CWORD-1]}"
#        case $prev in
#                '-h'|'--help'|'-v'|'--version')
#                        return 0
#                        ;;
#        esac
#        case $cur in
#                -*)
#                        OPTS="--favorites --kill-last-command --non-interactive --show-configuration --show-zsd-configuration --show-blacklist --version --help"
#                        COMPREPLY=( $(compgen -W "${OPTS[*]}" -- $cur) )
#                        return 0
#                        ;;
#        esac
#        compopt -o bashdefault
#        COMPREPLY=( $(compgen -c -- $cur) )
#        return 0
#}

_hstr ()
{
    hstr 2> /tmp/.hstr.tmp
    hstr_tmp=$(< /tmp/.hstr.tmp)
    __ehc "$hstr_tmp"
}

__ehc()
{
if
        [[ -n $1 ]]
then
        bind '"\er": redraw-current-line'
        bind '"\e^": magic-space'
        READLINE_LINE=${READLINE_LINE:+${READLINE_LINE:0:READLINE_POINT}}${1}${READLINE_LINE:+${READLINE_LINE:READLINE_POINT}}
        READLINE_POINT=$(( READLINE_POINT + ${#1} ))
else
        bind '"\er":'
        bind '"\e^":'
fi
}
complete -F _hstr hstr
complete -F _hstr hh

# eof
leapfog commented 1 year ago

While I still hope for hstr to be modified to work without CONFIG_LEGACY_TIOCSTI, kernel 6.2.x has been fixed, so re-enabling it at runtime is now possible:

# sysctl -w dev.tty.legacy_tiocsti=1 # fix hstr
dev.tty.legacy_tiocsti = 1

I only confirmed that with linux-6.2.6 today, but it might also have worked earlier.

karlovskiy commented 1 year ago

I can confirm that with 6.2.5-arch1-1 re-enabling sysctl -w dev.tty.legacy_tiocsti=1 is now also works.

Mte90 commented 1 year ago

I can confirm too on Debian sid (siduction) with 6.2.6-1 the hstr package from debian repository works again.

chrischmo commented 1 year ago

Re-enabling sysctl -w dev.tty.legacy_tiocsti=1 also works on OpenSUSE Tumbleweed (6.2.4-1-default) with hstr from the lemmy04 repo.

dvorka commented 1 year ago

@Mte90 @chrischmo @karlovskiy @leapfog @jakedane @papavlos @mmeier86 thank you all for detailed descriptions of the problem, root cause identification and proposed solutions!

I worked on a version of HSTR for Cygwin and WSL where TIOCSTI is not available in the past. If you are able to build HSTR and you use Bash, can you please try #481?

The PR just enables existing HSTR functionality to work w/o TIOCSTI w/ the new define. I did not used it in the past because it requires shell command (and many users will not (be able to) do such configuration).

Anyway if the PR will work for you, I will fix it also for zsh (which I use on 90% of my machines) and release a new version.

Do you think that it would make sense to release HSTR with a TIOCSTI parameter allowing to run version with or without TIOCSTI use? Is there any way how to safely detect TIOCSTI availability? Will it compile on systems with new kernel?

Thank you all your interest in HSTR and your help!

jakedane commented 1 year ago

@dvorka I built hstr with #481 and with LINUX_KERNEL_6 defined, I put the new config in my .bashrc and with that Ctrl+R works properly in Bash. Thank you!

I'm on Arch Linux with kernel 6.2.6. No issue compiling.

I don't know how to safely detect TIOCSTI is available. Probably too hacky and not portable but on Linux if sysctl -ne dev.tty.legacy_tiocsti prints 0 (zero) that means TIOCSTI is disabled.

dvorka commented 1 year ago

Preliminary TIOCSTI design:


.bashrc

# detect TIOCSTI
function is_tiocsti {
    if test -w /dev/tty && { stty -echo; echo -n a | tioctl TIOCSTI; stty echo; } >/dev/null 2>&1; then
        echo "TIOCSTI is supported by the kernel."
    else
        echo "TIOCSTI is not supported by the kernel."
    fi
}

# command binding
if is_tiocsti
then
    # if this is interactive shell, then bind hstr to Ctrl-r (for Vi mode check doc)
    if [[ $- =~ .*i.* ]]; then bind '"\C-r": "\C-a hstr -- \C-j"'; fi
    # if this is interactive shell, then bind 'kill last command' to Ctrl-x k
    if [[ $- =~ .*i.* ]]; then bind '"\C-xk": "\C-a hstr -k \C-j"'; fi
else
    if [[ $- =~ .*i.* ]]; then bind -x '"\C-r": "hstrcygwin"'; fi
fi 

Detect TIOCSTI from C:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <termios.h>
#include <sys/ioctl.h>

int main() {
   int fd;
   struct termios t;
   fd = open("/dev/tty", O_RDWR);
   if (fd < 0) {
      perror("open /dev/tty");
      exit(1);
   }
   if (tcgetattr(fd, &t) < 0) {
      perror("tcgetattr");
      exit(1);
   }
   if (!ioctl(fd, TIOCSTI, "a")) {
      printf("TIOCSTI supported\n");
   } else {
      printf("TIOCSTI not supported\n");
   }
   close(fd);
   return 0;
}
jakedane commented 1 year ago

The C code detects TIOCSTI correctly. I tested it both with dev.tty.legacy_tiocsti=0 and dev.tty.legacy_tiocsti=1.

The bash code does not detect it correctly. is_tiocsti always responds "TIOCSTI is supported by the kernel."

Edit: ah, removing the redirect to /dev/null I get tioctl: command not found.

dvorka commented 1 year ago

@jakedane thank you for testing the code! It really helped. As tioctl command does not have to be present on the system, I will always hstr - I will create a hstr parameter which will make the test (and use it in .bashrc/.zshrc).

dvorka commented 1 year ago

FYI working on the fix @ dev-2.7.0 branch.

hartwork commented 1 year ago

Hi!

I'd like to add some information regarding TIOCSTI ioctl that I didn't see mentioned above (from a quick look):

Best, Sebastian

dvorka commented 1 year ago

PROGRESS UPDATE: I apologize that the fix is not out yet (busy days). It was a bit painful process, but I finally have the solution for both Bash and Zsh which enables HSTR to work w/o TIOCSTI :relieved:

Zsh:

hstr_notiocsti() {
    zle -I
    { HSTR_OUT="$( { </dev/tty hstr ${BUFFER}; } 2>&1 1>&3 3>&- )"; } 3>&1;
    BUFFER="${HSTR_OUT}"
    CURSOR=${#BUFFER}
    zle redisplay
}
zle -N hstr_notiocsti
bindkey '\C-r' hstr_notiocsti

Bash:

function hstrnotiocsti {
    { HSTR_OUT="$( { </dev/tty hstr ${READLINE_LINE}; } 2>&1 1>&3 3>&- )"; } 3>&1;
    READLINE_LINE="${HSTR_OUT}"
    READLINE_POINT=${#READLINE_LINE}
}
if [[ $- =~ .*i.* ]]; then bind -x '"\C-r": "hstrnotiocsti"'; fi

Having the solution, I just need to polish it a bit and do the release. Stay tuned please.

krossekrabbe commented 1 year ago

Posted hstr_notiocsti() function does not work for me, still does nothing after selecting the history entry. Don't know if I am doing something wrong. I already verified that it is actually being executed on CTRL+R.

fedora 37, kernel 6.2.8-200.fc37.x86_64

sysctl -w dev.tty.legacy_tiocsti=1 is making it work.

dvorka commented 1 year ago

@krossekrabbe apologies for confusion - I shared the function, however, it will work with new version of HSTR. Anyway thank you for your interest in this issue!

dvorka commented 1 year ago

Fixed by aa14f032d4f39b38d7c7ccc042f18400760a3bd7 and released by https://github.com/dvorka/hstr/commit/f370a80e3d76cd74d0f3064e32ef86b0339a402f. Please report any problems!

EmJotGeh commented 1 year ago

The fix only seems to work to some extent. After entering the hstr_notiocsti() function in my .bashrc, the history can be opened via CTRL+R but the selected command now appears in the console and is not executed until ENTER is pressed. This corresponds more to the function of the right cursor key (->) - to display the command and to be able to change it if necessary.

cmonty14 commented 1 year ago

Imo the fix is not working as expected. This means I can display bash history and select any line item. However when I hit ENTER the line item will only be copied to console, but not executed.

I'm running Arch Linux with hstr version "3.1.0" (2023-04-18T08:50:00) and Linux Kernel 6.2.12-arch1-1. TIOCSTI is disabled (dev.tty.legacy_tiocsti = 0).

My understanding was that hstr should work /w and w/o TIOCSTI.

krossekrabbe commented 1 year ago

Yes can confirm on fedora, ENTER writes to console but not executes.

But I can live with that for now, close enough :joy: Thanks for the fix @dvorka

jakedane commented 1 year ago

My error I think. Where I wrote for the earlier patch "with that Ctrl+R works properly in Bash" I meant it works the same as my workaround, which has this behavior (which I actually prefer), and should have clarified with the patch Enter puts the command on the console and Enter is needed again to execute it. Sorry @dvorka if I put you on wrong footing here!

gowza commented 1 year ago

Is the version 3.1.0 supposed to fix selecting command from running hstr command directly as well (not using Ctrl + R)? If so, it is not working for me with bash on Arch Linux with kernel version 6.2.12-arch1-1. The interactive history shows and I can search, but selecting a command using tab or enter returns to the prompt with the selected command on its own line and the prompt line empty.

Enabling TIOCSTI with sysctl -w dev.tty.legacy_tiocsti=1 make it works. Although the selected command still print on its own line as well as populating the prompt line.

Thank you for the great work btw.

cmonty14 commented 1 year ago

Hello, I'm running Linux Kernel 6.3.1-arch2-1 hstr 3.1-1 and it works only as expected with dev.tty.legacy_tiocsti = 1.

My understanding is that latest hstr release should work w/o TIOCSTI, therefore I don't understand why this issue is closed.

GiulioCentorame commented 1 year ago

The interactive history shows and I can search, but selecting a command using tab or enter returns to the prompt with the selected command on its own line and the prompt line empty.

Same issue on Fedora 38 (running on kernel 6.2.14-300.fc38.x86_64)

palves commented 1 year ago

Should there be a separate bug for this "selecting command does not execute directly" issue? I ask because this bug is closed.

Gooberpatrol66 commented 1 year ago

Hello, I'm running Linux Kernel 6.3.1-arch2-1 hstr 3.1-1 and it works only as expected with dev.tty.legacy_tiocsti = 1.

My understanding is that latest hstr release should work w/o TIOCSTI, therefore I don't understand why this issue is closed.

Same here, it was fixed for a while and then it broke again. kernel 6.3.9

quicktrick commented 1 year ago

Void Linux, kernel 6.3.10, hstr does not work.

smahm006 commented 1 year ago

Confirming on Debian Sid kernel 6.3.0-1-amd64 has the same issue. sysctl -w dev.tty.legacy_tiocsti=1 does fix the issue.

rivenirvana commented 1 year ago

This needs to be reopened.

jakedane commented 1 year ago

Several comments above saying hstr doesn't work. It's not clear what "doesn't work" means.

Does it mean hstr doesn't work at all without setting dev.tty.legacy_tiocsti=1? And if so with which shell? Or does it mean hstr works but Enter needs to be pressed twice to run a command from history? First Enter puts the command on the prompt, second Enter runs it. In either case I think that should be a new issue.

I'm on Arch Linux with kernel 6.4.4 and hstr works for me in Bash. Enter needs to be pressed twice to run a command from history which is different from with setting dev.tty.legacy_tiocsti=1 but it does work.

Gooberpatrol66 commented 1 year ago

Does it mean hstr doesn't work at all without setting dev.tty.legacy_tiocsti=1?

The same behavior as before the fix was applied

And if so with which shell?

bash

Or does it mean hstr works but Enter needs to be pressed twice to run a command from history?

No, that's not it

jakedane commented 1 year ago

@Gooberpatrol66 which distro are you on? Which terminal are you using? Which hstr version and how does your configuration in .bashrc differ from the recommended:

# HSTR configuration - add this to ~/.bashrc
alias hh=hstr                    # hh to be alias for hstr
export HSTR_CONFIG=hicolor       # get more colors
shopt -s histappend              # append new history items to .bash_history
export HISTCONTROL=ignorespace   # leading space hides commands from history
export HISTFILESIZE=10000        # increase history file size (default is 500)
export HISTSIZE=${HISTFILESIZE}  # increase history size (default is 500)
# ensure synchronization between bash memory and history file
export PROMPT_COMMAND="history -a; history -n; ${PROMPT_COMMAND}"
function hstrnotiocsti {
    { READLINE_LINE="$( { </dev/tty hstr ${READLINE_LINE}; } 2>&1 1>&3 3>&- )"; } 3>&1;
    READLINE_POINT=${#READLINE_LINE}
}
# if this is interactive shell, then bind hstr to Ctrl-r (for Vi mode check doc)
if [[ $- =~ .*i.* ]]; then bind -x '"\C-r": "hstrnotiocsti"'; fi
export HSTR_TIOCSTI=n

I'm using hstr 3.1, bash 5.1.016 and Linux 6.4.4 on Arch Linux. That works. I tried with Gnome Terminal, Console and Black Box. My .bashrc uses the recommended configuration.

I also tried hstr 3.0 on Fedora 38 and after manually fixing hstrnotiocsti in .bashrc, that also works.

Gooberpatrol66 commented 1 year ago

oh, i didn't see the bashrc changed

eleius commented 1 year ago

I forgot I was still using the sysctl "fix", so I removed it and now hstr doesn't work for me.

I'm using hstr 3.1.0 on arch linux, kernel 6.5.5, and my bashrc has the same lines as in jakedane's comment above.

I can search history but pressing enter doesn't select the entry, it only echo'es it. If I set dev.tty.legacy_tiocsti=1 it works.

jakedane commented 1 year ago

I can search history but pressing enter doesn't select the entry, it only echo'es it.

You have to press Enter twice. 1st Enter puts the selected history item on the bash prompt, 2nd Enter runs it.

If any expect hstr to work differently (that 1st Enter runs the command directly), I think that needs to be a separate issue.

jessienab commented 1 year ago

I can search history but pressing enter doesn't select the entry, it only echo'es it.

You have to press Enter twice. 1st Enter puts the selected history item on the bash prompt, 2nd Enter runs it.

If any expect hstr to work differently (that 1st Enter runs the command directly), I think that needs to be a separate issue.

@eleius another suggestion is to select your command, use TAB to bring it to your prompt, and then press enter; removing the need to hit Enter twice.

eleius commented 1 year ago

@jakedane @jasonnab I've already tried Tab+Enter and Enter+Enter, but for some reason when I press either Tab or Enter, the selected history item is just displayed as when you type the echo command (hstr doesn't bring it to the bash prompt.) I've tried deleting my current .bashrc and only placing the hstr lines in it, but still same issue. Don't know why it doesn't work.

quicktrick commented 1 year ago

@eleius, don't be upset. I have hstr behaving exactly the same way on Void Linux 6.3.13.

Mte90 commented 11 months ago

any update for this issue? when it is planned a new release?

jakedane commented 11 months ago

@Mte90 hstr 3.1.0 fixed the issue. Currently with bash 5.2.21 in gnome-terminal 3.50.1 (vte 0.74.1) on Arch Linux, kernel 6.6.3, with the output of hstr --show-configuration in the .bashrc file, that works without problems. The kernel is not compiled with CONFIG_LEGACY_TIOCSTI set and dev.tty.legacy_tiocsti is also not set.

If it doesn't work for you, what specifically are you using?