search

akinomyoga / ble.sh

Bash Line Editor―a line editor written in pure Bash with syntax highlighting, auto suggestions, vim modes, etc. for Bash interactive sessions.
BSD 3-Clause "New" or "Revised" License
2.53k stars 81 forks source link

[commit ffac420, pquery, portageq] Error messages from auto-complete #487

Open blackteahamburger opened 3 weeks ago

blackteahamburger commented 3 weeks ago 
ble/widget/display-shell-version:
GNU bash, version 5.2.32(1)-release (x86_64-pc-linux-gnu) [Gentoo Linux]
ble.sh, version 0.4.0-devel4+70b89e5e (noarch) [git 2.46.0, GNU Make 4.4.1, GNU Awk 5.3.0, API 4.0]
bash-completion, version 2.14.0 (hash:480ffcc6a751e55621ec526eb5dea7a0d86d9e72, 17877 bytes) (noarch)
locale: LANG=C
terminal: TERM=xterm-256color wcwidth=auto-auto/15.1-2+ri, konsole:220380 (1;115;0)
options: +extglob +histappend -hostcomplete +inherit_errexit +no_empty_cmd_completion

Input something=, some error messages will be shown, e.g. (input LANG=):

$ LANG=bash: comp_words: bad array subscript
bash: comp_words: bad array subscript
bash: comp_words: bad array subscript

And another strange bug with pquery (See https://pkgcore.github.io/pkgcore/man/pquery.html#pquery) (maybe there's more commands with the same problem)

Input pquery something, some error messages will be shown, e.g. (input pquery s):

$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

But no error message will be shown if the current working directory is a git repository.

akinomyoga commented 3 weeks ago

Input something=, some error messages will be shown, e.g. (input LANG=):

Thank you for the report. Confirmed. This is a regression introduced in commit ffac4205.

And another strange bug with pquery (See https://pkgcore.github.io/pkgcore/man/pquery.html#pquery) (maybe there's more commands with the same problem)

$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

This must be caused in the programmable completions called from auto-complete.

$ ble-detach
[ble: detached]
Please run `stty sane' to recover the correct TTY state.
$  stty sane;[RET]
$ pquery s[TAB] <-- Does the problem reproduce here?
akinomyoga commented 3 weeks ago

Input something=, some error messages will be shown, e.g. (input LANG=):

Thank you for the report. Confirmed. This is a regression introduced in commit ffac420.

I pushed a fix 9270b529cf9969865bbef7944b9ff8e317306ad5 for this problem.

akinomyoga commented 3 weeks ago 
$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

This is an issue in the upstream. This line is always executed without redirecting stderr, which is the problem.

blackteahamburger commented 3 weeks ago
  • Q1: Does the problem persist after you set bleopt complete_auto_complete=?

No. Error msgs disappeared.

  • Q2: Does the problem happen in the detached state of ble.sh on attempting the TAB completion?

Yes with the same error msg.

akinomyoga commented 3 weeks ago

Thank you, so it is not the issue with ble.sh. It's an upstream bug in pkgcore, which provides the pquery completion.

blackteahamburger commented 3 weeks ago

Thank you for finding the source of the bug and the PR!

blackteahamburger commented 2 weeks ago

There is one more command with error messages from auto-complete.

Input portageq envvar something (see here for bashcomp), some error messages will be shown, e.g.:

$ portageq envvar s!!! "--ask" should only be used in a terminal. Exiting.
!!! "--ask" should only be used in a terminal. Exiting.
s

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

Related: https://forums.gentoo.org/viewtopic-t-1068540-start-0.html

  • Q1: Does the problem persist after you set bleopt complete_auto_complete=?

No. Error msgs disappeared.

  • Q2: Does the problem happen in the detached state of ble.sh on attempting the TAB completion?

No.

$ portageq envvar[TAB]
Display all 169 possibilities? (y or n)[y]
ABI                                      CHOST_x86                                GSETTINGS_BACKEND                        OFFICE_IMPLEMENTATION                    POSTGRES_TARGETS
ABI_X86                                  CLEAN_DELAY                              GUILE_SINGLE_TARGET                      PATH                                     PROFILE_ONLY_VARIABLES
ACCEPT_KEYWORDS                          COLLECTD_PLUGINS                         GUILE_TARGETS                            PHP_TARGETS                              PYTHONDONTWRITEBYTECODE
ACCEPT_LICENSE                           COLLISION_IGNORE                         INFOPATH                                 PKGDIR                                   PYTHON_SINGLE_TARGET
ACCEPT_PROPERTIES                        CONFIG_PROTECT                           INPUT_DEVICES                            PORTAGE_ARCHLIST                         PYTHON_TARGETS
ACCEPT_RESTRICT                          CONFIG_PROTECT_MASK                      IUSE_IMPLICIT                            PORTAGE_BIN_PATH                         RESUMECOMMAND
...

I believe this is an inner bug because there's no related bug report in Gentoo's Bugzilla.

akinomyoga commented 2 weeks ago

I believe this is an inner bug because there's no related bug report in Gentoo's Bugzilla.

This is a compatibility issue, or just the completion setting is "not yet supported" by ble.sh. Since Bash doesn't offer the feature of autosuggestions, existing completion settings are designed to be used with the TAB completion. They are not designed to be used by autosuggestions. If you say this is a bug, the entire auto-complete feature based on the existing completion settings is a bug, but I do not want to remove the auto-complete feature based on the completion settings. It is impractical to prepare the dedicated auto-completion settings for all the commands in the wild by ourselves, so it is still beneficial to use the existing completion settings.

The present case is related to something that ble.sh disables for the completion settings. Some completion settings, on the user's attempt at the TAB completion, try to open a TUI window and wait for the user's input from TTY/PTY on stdin. Although it seems like hijacking of the TAB completion of Readline, Bash technically doesn't prohibit or discourage such a completion setting. If such a completion setting is naively called from the auto-complete feature, it would open the TUI window on every keystroke of the user and randomly steals the user's input. This would make the shell almost unusable. At worst, such a completion setting called in an abnormal setup by ble.sh would block the session, and you will lose control of the session. It is almost impossible for ble.sh to robustly judge whether a given completion setting would cause a problem in auto-complete without prior knowledge, so it is not possible to block such a completion setting before calling it. Instead, ble.sh tries to call the completion setting with stdin redirected to /dev/null, expecting such a completion setting would immediately fail. This probably works fine, but this is merely an expectation. The existence of completion settings that do not work with ble.sh's auto-complete is anticipated, and we are going to patch such completion settings one by one to officially "support" them. That is my choice. There is no better way other than disabling auto-complete based on the completion settings. Fortunately, most of the completion settings do not try to access TTY/PTY, so it seems we only need to patch finite number of completion settings.

It should also be noted that even for the TAB completion, other limitations exist in the custom terminal state set up by ble.sh and also in the signal handling while calling the completion settings. For this reason, ble.sh actually disables stdin even for the TAB completion by default. ble.sh wants to limit the access to the TTY to the completion settings patched by ble.sh. This is also related to the special setup needed for fzf's completion settings notified in ble.sh's README. The fzf completion is a typical completion that opens a TUI window.

akinomyoga commented 2 weeks ago

So I only see env -i PATH="${PATH}" emerge -v --info in the completion settings, where I don't see the option --ask or -a. Does that try to access TTY or attempt to read from stdin? Does running the following command in the command line produce the same message?

$ env -i PATH="${PATH}" emerge -v --info </dev/null
akinomyoga commented 2 weeks ago

Neither option -v nor --info seems to imply --ask, but it seems possible to globally configure the default options in /etc/portage/make.conf. Do you have --ask in EMERGE_DEFAULT_OPTS?

blackteahamburger commented 2 weeks ago

Neither option -v nor --info seems to imply --ask, but it seems possible to globally configure the default options in /etc/portage/make.conf. Do you have --ask in EMERGE_DEFAULT_OPTS?

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

I believe it's quite common among Gentoo users because the default behavior of emerge is that it does not ask anything and run by itself. It didn't cause any problems before.

akinomyoga commented 2 weeks ago

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

That means that --ask is actually not the flag requested by the completion setting. The completion setting can work without the --ask flag and is designed so by default.

I believe it's quite common among Gentoo users because the default behavior of emerge is that it does not ask anything and run by itself.

Yeah, and in fact, that is suggested by emerge - Gentoo Wiki. However, I see some inconsistency in the design of the behavior of --ask and the description there.

These two points appear to conflict with each other by design. We cannot safely use the emergy command in scripts. In particular, it would be a problem that --ask outputs the error message even when the specific usage of emerge in a script never requires confirmation.

[^1]: If it's not intended to be used in scripts at all and is required to be run only in the interactive shell sessions, it should always print the error message whenever it is called in scripts without the terminal. However, the emergy command prints the error message only when --ask is printed. In addition, the completion setting is technically not an interactive command, yet emerge -v --info is called. So, the use in scripts is also within a scope of some emerge calls.

I wondered if there weren't any problems with the --ask behavior. A quick search finds many problems caused by this behavior:

This must be a design issue of --ask when specified in EMERGE_DEFAULT_OPTS. In the first link above, specifying --ask=n or --ignore-default-opts at the script side is suggested as a solution. In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

It didn't cause any problems before.

That is just because your situation has changed.

blackteahamburger commented 2 weeks ago

  • A question is whether the call of emerge -v --info, with --ask supplied by the default setting, has a chance to ask a user to respond. Option --info seems to print just information, and I guess it wouldn't do any destructive operation, so the user's confirmation about whether it should print the information is unlikely to be needed/useful.

    • If the answer is no, we may ask if they can specify --ask=n or redirect stderr for the call of emerge -v --info.

    • If the answer is yes, the next question is whether it would be expected and useful in the middle of the completion. I don't see any use case. If --info would try to do destructive operations, the answer to the confirmation should always be "no" in the middle of completions, so the confirmation could be skipped.

    • If the answer is no, --ask=n should actually be specified to the call of emergy -v --info.

    • If the answer is yes, what is that?

It simply print the information, nothing else is done, and without any confirmation even with --ask.

In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

I don't know what happened 18 years ago but --ignore-default-opts now works well on emerge --sync.

akinomyoga commented 2 weeks ago

  • A question is whether the call of emerge -v --info, with --ask supplied by the default setting, has a chance to ask a user to respond. Option --info seems to print just information, and I guess it wouldn't do any destructive operation, so the user's confirmation about whether it should print the information is unlikely to be needed/useful.

    • If the answer is no, we may ask if they can specify --ask=n or redirect stderr for the call of emerge -v --info.

It simply print the information, nothing else is done, and without any confirmation even with --ask.

Thank you for the information. Then, I would probably ask them if they can include --ask=n in the call of emerge -v --info.

In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

I don't know what happened 18 years ago but --ignore-default-opts now works well on emerge --sync.

Thank you also for this information.

Just to confirm the behavior, what are the results of the following commands (with --ask specified to EMERGE_DEFAULT_OPTS?

$ env -i PATH="${PATH}" emerge -v --info < /dev/null
$ env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
$ env -i PATH="${PATH}" emerge -v --info --ignore-default-opts < /dev/null

Maybe specifying --ignore-default-opts in the completion setting should be safer than just specifying --ask=n because there might be other options that can interfere with it. Do you think specifying --ignore-default-opts would make sense? Or is there a possibility that it affects the set of completions for envvar in an unexpected way? Sorry for asking many things, but I'm not familiar with emerge because I'm not a user of Gentoo.

blackteahamburger commented 2 weeks ago 
$ env -i PATH="${PATH}" emerge -v --info < /dev/null
!!! "--ask" should only be used in a terminal. Exiting.
$ env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
...
ABI="amd64"
ABI_X86="64"
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
ACCEPT_PROPERTIES="*"
ACCEPT_RESTRICT="*"
ADA_TARGET="gcc_12"
APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias"
ARCH="amd64"
AUTOCLEAN="no"
BINPKG_COMPRESS="zstd"
BINPKG_FORMAT="xpak"
BINPKG_GPG_SIGNING_BASE_COMMAND="/usr/bin/flock /run/lock/portage-binpkg-gpg.lock /usr/bin/gpg --sign --armor [PORTAGE_CONFIG]"
BINPKG_GPG_SIGNING_DIGEST="SHA512"
BINPKG_GPG_VERIFY_BASE_COMMAND="/usr/bin/gpg --verify --batch --no-tty --no-auto-check-trustdb --status-fd 2 [PORTAGE_CONFIG] [SIGNATURE]"
BINPKG_GPG_VERIFY_GPG_HOME="/etc/portage/gnupg"
BOOTSTRAP_USE="unicode pkg-config split-usr xml python_targets_python3_12 python_single_target_python3_12 gil multilib zstd cet systemd sysv-utils udev"
BROOT=""
CALLIGRA_FEATURES="karbon sheets words"
CAMERAS=""
CARGO_PROFILE_RELEASE_CODEGEN_UNITS="1"
CARGO_PROFILE_RELEASE_LTO="fat"
CARGO_PROFILE_RELEASE_OPT_LEVEL="3"
CARGO_PROFILE_RELEASE_PANIC="abort"
CARGO_PROFILE_RELEASE_STRIP="symbols"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=auto -fisolate-erroneous-paths-attribute -pipe"
CFLAGS_amd64="-m64"
CFLAGS_x32="-mx32"
CFLAGS_x86="-m32 -mfpmath=sse"
CHOST="x86_64-pc-linux-gnu"
CHOST_amd64="x86_64-pc-linux-gnu"
CHOST_x32="x86_64-pc-linux-gnux32"
CHOST_x86="i686-pc-linux-gnu"
CLEAN_DELAY="5"
COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog"
COLLISION_IGNORE="/boot/dtbs/* /lib/modules/*"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d"
CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3"
CXXFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=auto -fisolate-erroneous-paths-attribute -pipe -fforce-emit-vtables -fstrict-vtable-pointers -fwhole-program-vtables"
DEFAULT_ABI="amd64"
DISTDIR="/var/cache/distfiles"
EDITOR="nano"
ELIBC="glibc"
EMERGE_DEFAULT_OPTS="-av" # note here
...
$ env -i PATH="${PATH}" emerge -v --info --ignore-default-opts < /dev/null
... # exactly the same output as env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
EMERGE_DEFAULT_OPTS="-av" # same here
...

So --ignore-default-opts is good.

akinomyoga commented 2 weeks ago

According to Manpage of EMERGE, there seems to be an option --prefix that affects the environment variable EPREFIX. I'm afraid that it might affect the list of envvars when --prefix is specified in the default options. What are the results of the following commands (/tmp can be adjusted if necessary)?

$ env -i PATH="${PATH}" emerge -v --info               | grep -o '^EPREFIX'
$ env -i PATH="${PATH}" emerge -v --info --prefix=/tmp | grep -o '^EPREFIX'
blackteahamburger commented 2 weeks ago

It really differs:

$ env -i PATH="${PATH}" emerge -v --info
...
EPREFIX=""
EROOT="/"
ESYSROOT="/"
...
$ env -i PATH="${PATH}" emerge -v --info --prefix=/tmp
...
EPREFIX="/tmp"
EROOT="/tmp/"
ESYSROOT="/tmp/"
...
akinomyoga commented 2 weeks ago

Thank you. So EPREFIX etc. exist even if --prefix is not specified. The differences in the values don't matter for the completion setting of portageq because any strings after = are anyway stripped on this line. Then, I think we don't have to care about this point for --ignore-default-opts.