Implement zsh-like preexec and precmd hook

andychu commented 4 years ago

Running starship.rs

https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Running.20starship.2Ers

andychu commented 4 years ago

code:

https://github.com/starship/starship/blob/master/src/init/starship.bash

andychu commented 4 years ago

Not unexpected: it has some weird hacks around bash's runtime:

# We use PROMPT_COMMAND and the DEBUG trap to generate timing information. We try
# to avoid clobbering what we can, and try to give the user ways around our
# clobbers, if it's unavoidable. For example, PROMPT_COMMAND is appended to,
# and the DEBUG trap is layered with other traps, if it exists.

# A bash quirk is that the DEBUG trap is fired every time a command runs, even
# if it's later on in the pipeline. If uncorrected, this could cause bad timing
# data for commands like `slow | slow | fast`, since the timer starts at the start
# of the "fast" command.

# To solve this, we set a flag `PREEXEC_READY` when the prompt is drawn, and only
# start the timer if this flag is present. That way, timing is for the entire command,
# and not just a portion of it.

akinomyoga commented 4 years ago

https://github.com/starship/starship/blob/master/src/init/starship.bash

I would like to point out that the same technique with DEBUG trap is used in iTerm2-Shell integration in macOS (iterm2_shell_integration.bash) which borrows the code of rcaloras/bash-preexec (bash-preexec.sh). The technique is outlined in a StackOverflow answer. According to the answer, this trick was initially introduced by Glyph Lefkowitz (preexec.bash). It seems that iTerm2-Shell integration is pretty popular as I have received related Issues in ble.sh a few times.

andychu commented 4 years ago

Thanks for the info! I just tested out the DEBUG behavior in bash and I find it confusing. help trap (in 4.4) says it only executes on simple command.

The manual says it executes on a few more constructs -- like for loops, case, etc. which I observed.
It doesn't seem to go into functions? That is odd. It only works at the top level?
And yes behavior in pipelines is a little confusing. The {echo a; echo b; } | wc -l case seems inconsistent with the others. I guess it's not inherited in the first subshell or something.

So I would like it if we can implement something more narrow and well-specified, maybe zsh precmd and preexec makes more sense. And then applications can use those directly.

I'm not sure yet exactly what we'll do -- the use cases help decide that. But right now I don't feel like copying bash quirks :)

andychu commented 4 years ago

In other words if you can think of a better mechanism that most shell scripts can use, I'm interested :)

It seems like they hack around the mechanism anyway to get what they want. So we should just implement what they want rather than copy bash.

They can do something like:

if test -n "$OIL_VERSION"; then
  trap myhook OIL_SIMPLE_COMMAND
  trap pipelinehook OIL_PIPELINE
else
  trap bashhook DEBUG
fi

or something like that

akinomyoga commented 4 years ago

Inheritance of `DEBUG` trap

It doesn't seem to go into functions? That is odd. It only works at the top level?

It's configurable (as documented in the Bash Reference Manual 3.3). DEBUG trap is inherited by functions when set -o functrace is specified. Or, one can use declare -ft function_name to enable DEBUG inheritance for each function. Also, when a DEBUG trap is set in a function call, the trap is also effective in callers.

$ trap 'echo "[DEBUG] $BASH_COMMAND"' DEBUG
$ f1() { echo hello; }
$ f1
[DEBUG] f1
hello

set -o functrace

$ set -o functrace
[DEBUG] set -o functrace
$ f1
[DEBUG] f1
[DEBUG] f1
[DEBUG] echo hello
hello
$ set +o functrace
[DEBUG] set +o functrace
$ f1
[DEBUG] f1
hello

declare -ft f1

$ declare -ft f1
[DEBUG] declare -tf f1
$ f1
[DEBUG] f1
[DEBUG] f1
[DEBUG] echo hello
hello
$ declare -f +t f1
[DEBUG] declare -f +t f1
$ f1
[DEBUG] f1
hello

The trap call in a function call affects callers:

$ f1() { trap 'echo "[DEBUG by f1] $BASH_COMMAND"' DEBUG; echo f1; }
$ f2() { f1; echo f2; }
$ f3() { f2; echo f3; }
$ f3
[DEBUG by f1] echo f1
f1
[DEBUG by f1] echo f2
f2
[DEBUG by f1] echo f3
f3

Better mechanism?

In other words if you can think of a better mechanism that most shell scripts can use, I'm interested :)

It seems like they hack around the mechanism anyway to get what they want.

Yes, starship, iterm2_shell_integration and bash-preexec hacks the DEBUG trap to mimic the Zsh's preexec hook. It is not the intended usage of the DEBUG trap. The preexec hook is just a hook for the event that a user interactively executes the command (by hitting RET, etc.).
The usage in ble.sh is another type of hack. It wants to cancel the current execution of nested function calls below a certain depth. This is actually an emulation of try ... catch ... finally ....
The original usage for DEBUG trap is to provide an API that can be used by debugger such as bashdb. I guess set -o functrace is used to switch between Step In and Step Over, the return status of DEBUG trap handler can be used to Jump Over a command in execution, etc.

In this sense, the DEBUG trap can be regarded as a low-level hook for controlling the execution engine which enables various hacks. These usages can be replaced by the following more high-level APIs. But I wouldn't be surprised if there are still other hacks by the DEBUG trap which cannot be covered by the following high-level APIs:

preexec hook
try ... catch ... finally
high-level Debugger API

Edit: I guess #708 (ERR, BASH_ARGV, extdebug) is also related. I found a mention on try/catch at #477, but it is not discussed deeply.

andychu commented 4 years ago

Thank you, that's very helpful! I think we can start with some hooks for a few use cases and then expand them if people have more.

I'll think about it but if you have an idea you want to implement for ble.sh or anything else, that would be welcome too.

About try/catch: I think we won't have it now, although I don't have the error handling story for expressions completely settled.

Details: Originally Oil was going to reuse parts of the Python interpreter, basically because I like Python's data structures (dict, list, tuple, float, int, string). There is a lot of subtlety and power there, like comparing tuples with > as in issue #683 .

And Python expressions can raise exceptions, so I felt like we needed try/catch.

But I decided not to reuse any of CPython because:

1) I found that the languages don't compose. You have to build a lot of "bridges" between them. Hard to explain but Python composes in a lot of ways like iter() and list() and dict() (taking tuples, etc.) Iterators are a core part of the model, then generators, etc. Everything is very tightly integrated. Which works well but doesn't integrate with a shell.

2) There are at least 10-20 things I would want to fix (should floats be hashable, Python 3 backports like > is an error across different types, and dictionary insertion order), and that will take a long time.

So I think that there are just keywords that take a expressions like var and const, and when those raise exceptions, you can't catch them. They just result in the var or const statement returning 1, and you can handle that in the norrmal shell way.

shopt --unset errexit {
  var x = 1 / 0
  echo $?   # status 1.  Normally that is fatal
}

andychu commented 3 years ago

DEBUG hook mentioned here: https://old.reddit.com/r/programming/comments/lccfpq/why_create_a_new_unix_shell_2021/gm2c7io/

Rycieos commented 3 years ago

DEBUG hook mentioned here: https://old.reddit.com/r/programming/comments/lccfpq/why_create_a_new_unix_shell_2021/gm2c7io/

Guy from that thread here.

To give more context, we also use Zsh's add-zsh-hook of types of both preexec and precmd. A similar functionality in Osh would be simple to support, and would be preferable, as we need to do some hacky tricks to work around the quirks in Bash DEBUG (similar to what the previously mentioned bash-preexec does).

andychu commented 3 years ago

Another user ran into this trying to run starship: https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/A.20list.20of.20feedback

andychu commented 3 years ago

This project to emulate zsh hooks in bash: https://github.com/rcaloras/bash-preexec

mentioned here, so there is demand:

https://blog.warp.dev/how-warp-works/

i.e. we could have the zsh-like hooks directly.

andychu commented 1 year ago

@Melkor333 also ran into this, time to implement it

https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Features.20that.20Real.20Plugins.20Use

andychu commented 1 year ago

We did trap DEBUG in #1546 , and now I understand zsh precmd and preexec, and I think that is desirable

So leaving this bug for that

oils-for-unix / oils