ble.sh thinks coproc are jobs on C-d, native bash does not

[bkerin]

$ echo "$BLE_VERSION"
0.4.0-devel3+cc852dc $ echo "$BASH_VERSION ($MACHTYPE)"
5.0.17(1)-release (x86_64-pc-linux-gnu)

I think the bash behavior is better in this respect, since coproc are quite different that job-controlled jobs and should probably just silently get cleaned up. Even better would be an option in bash coproc to control this behavior and also maybe hide from jobs, but if you had to pick one I'd follow the bash behavior.

ble.sh behavior:

$ jobs 504 [1]+ Running coproc schwbrs { cat; } & $ [ble: There are remaining jobs. Use "exit" to leave the shell.] <--- On C-d

[akinomyoga]

I think it is technically possible to mimic Bash's behavior so that C-d skips the check for coproc jobs. I'd consider it later.

I think the bash behavior is better in this respect, since coproc are quite different that job-controlled jobs and should probably just silently get cleaned up.

As far as I know, Bash doesn't clean up coproc processes on the session exit, though the coproc processes might receive errors in reading stdin or outputting to stdout.

[akinomyoga]

As I have replied in the help-bash mailing list, there are already some codes in contrib/histdb.bash. For 2 and 3, maybe I can turn them into a reusable function that can be used to create an arbitrary background process with mutual pipe connections to the main shell.

[akinomyoga]

I now tried to check the detailed behavior of Bash, but Bash doesn't seem to change the behavior of C-d for normal background jobs and coproc jobs. When the coproc process is suspended, Bash doesn't exit with C-d:

$ LANG=C bash --norc
$ coproc a { sleep 100; }
[1] 1620290
$ fg
coproc a { sleep 100; }
<C-z>^Z
[1]+  Stopped                 coproc a { sleep 100; }
$<C-d>
exit
There are stopped jobs.
$

I think the behavior difference that you noticed is not the difference between the normal background jobs and coproc jobs. I think it is the difference between running background jobs and suspended background jobs. While Bash only checks for suspended background jobs, ble.sh intentionally checks both types of jobs before attempting to exit the session with C-d.

If Bash exits without properly finishing the running jobs, those jobs will continue to run after the session is ended. This is also the case for coproc processes, though in some cases the coproc processes might fail with pipe errors or other types of crashes. On the other hand, as C-d is also used to delete the character on the current position, one could mistakenly hit C-d on an empty command line, and the Bash can exit unintendedly. I think this is a problem, so ble.sh intentionally blocks the exiting by C-d when there are any types of background jobs.

[bkerin]

You're right about background vs suspended and therefore again ble.sh behavior looks better even before the C-d consideration.

There's also disown command that I didn't know about to hide from 'jobs' etc. These coproc don't survive past termination of parent process I expect due to default SIGPIPE dispensation, which seems like enough given that purpose of coproc seems to be convenient setup of piped child.

So it looks like the only remaining question is about the status of coproc implementation with respect to multiple coproc. The mailing list response on this contradicts some info elsewhere on this and I'm going to see what's going on with recent bash build options.

[akinomyoga]

This is a reply to the blesh-related comments in the help-bash mailing list.

From https://lists.gnu.org/archive/html/help-bash/2023-03/msg00076.html Yes, you are exactly right, and in fact it's ble.sh that's getting it I think. It also eats STDERR in general after:

[[ ${BLE_VERSION-} ]] && ble-attach

Yes. If you would like to output anything to stderr, such commands should be put before ble-attach. This is the reason that the above line needs to be put at the bottom of ~/bashrc.

I know this was mentioned in ble.sh docs but I've still managed to confuse myself with it a couple times now with e.g. typos in bleopt that seem to need to follow ble-attach. Is there a way to avoid this effect?

Some of the options are dynamically defined by modules, which might not be yet defined by the module. The recommended way is to use the proper hooks (blehook/eval-after-load NAME HOOK or ble/util/import/eval-after-load MODULE HOOK).

I've thought of putting all the config that seems to need to follow ble-attach in hooks or something then make ble-attach the very last line of ~/.bashrc but I'm not sure it would work.

Yes, that is actually the proper way. If you could give me the list of the option names that you do not know which hooks to put them in, I can tell you the corresponding hook names. Or you might just grep bleopt/declare to identify the module name defining the option name and set up a hook as ble/util/import/eval-after-load MODULE_NAME HOOK_NAME.

But if you would like to forcibly define the options before loading the corresponding modules, you can actually define them by the form bleopt name:=value (notice that := is used in place of =).

[akinomyoga]

https://github.com/akinomyoga/ble.sh/issues/302#issuecomment-1462977627 by @akinomyoga [...] For 2 and 3, maybe I can turn them into a reusable function that can be used to create an arbitrary background process with mutual pipe connections to the main shell.

I abstracted the codes in contrib/histdb and added a new module util.bgproc in commit 7803305f1afd9476e372c0d9c5c4357153bd3a73. I added an example usage at contrib/config/github302-perlre-server.bash.

$ ble-import config/github302-perlre-server
ble/contrib/config:github30: perlre-server (1788726) has started.
$ ble/contrib/config:github302/perlre-match '<a>([^<>]*)</a>' " <a>123456789101112</a> "
$ echo $ret
15

Explanations are written in the code comments of contrib/config/github302-perlre-server.bash. Mode retailed usage of ble/util/bgproc is decribed in the code comments of lib/util.bgproc.sh.

[akinomyoga]

So it looks like the only remaining question is about the status of coproc implementation with respect to multiple coproc. The mailing list response on this contradicts some info elsewhere on this and I'm going to see what's going on with recent bash build options.

@bkerin Do you have any updates on this? If I correctly understand it, is it a question/issue about the upstream Bash?

[bkerin]

I haven't learned anything new about the implementation in bash, I tried building bash with the option for multiple coproc support but I must have done it wrong because same error. A single coproc is ok for me at the moment as I don't use them for anything else. If I get around to adding to blesh contrib I'll try to use your setup above.

This issue can be closed.

[akinomyoga]

Thank you for the reply.

Hmm, I now tried the option for multiple coproc, but it seems to work in my environment both with the devel branch of Bash and with the release version of bash-5.2. You need to edit config-top.h and rebuild Bash to enable the multiple coproc support. Maybe another possibility is that you tested the behavior with the system Bash instead of the newly installed Bash. The path of the currently running Bash can be checked with $BASH (but not $SHELL, etc.).

[bkerin]

This is a reply to the blesh-related comments in the help-bash mailing list.

From https://lists.gnu.org/archive/html/help-bash/2023-03/msg00076.html Yes, you are exactly right, and in fact it's ble.sh that's getting it I think. It also eats STDERR in general after:

[[ ${BLE_VERSION-} ]] && ble-attach

Yes. If you would like to output anything to stderr, such commands should be put before ble-attach. This is the reason that the above line needs to be put at the bottom of ~/bashrc.

I know this was mentioned in ble.sh docs but I've still managed to confuse myself with it a couple times now with e.g. typos in bleopt that seem to need to follow ble-attach. Is there a way to avoid this effect?

Some of the options are dynamically defined by modules, which might not be yet defined by the module. The recommended way is to use the proper hooks (blehook/eval-after-load NAME HOOK or ble/util/import/eval-after-load MODULE HOOK).

I've thought of putting all the config that seems to need to follow ble-attach in hooks or something then make ble-attach the very last line of ~/.bashrc but I'm not sure it would work.

Yes, that is actually the proper way. If you could give me the list of the option names that you do not know which hooks to put them in, I can tell you the corresponding hook names. Or you might just grep bleopt/declare to identify the module name defining the option name and set up a hook as ble/util/import/eval-after-load MODULE_NAME HOOK_NAME.

I got around to doing this and it works nicely, no more silent typo failures. One feature I wonder about though is a sort of after-everything type hook in which casual users could simply place their entire ble.sh configuration. The recommended way would become something like:

source ~/.local/share/blesh/ble.sh --noattach
[existing .bashrc]
function my/blesh-setup
blehook/eval-after-load after-everything-module my/blesh-setup
[[ ${BLE_VERSION-} ]] && ble-attach

This spares the end user from having to track where things come from. I realize after-everything can be problematic (e.g. systemd have never supported) because everyone wants to be in it, but from the perspective of end users who don't want to know too much it's nice, and the only challenge is keeping devs out of it :)

[akinomyoga]

How do you define after-everything? There are different levels of delayed initializations, idle processing, and background processing.

If you want to define a hook that is called after all the modules are loaded, there is still a subtlety. Some modules can be dynamically loaded in the idle state. Some modules can be loaded on demand after a long time, Some modules can be loaded by the user by running the ble-import command. There is no real end or deadline in the module loading. If you need to make sure that the code is run after particular modules are loaded, the code should call the corresponding ble-import at the beginning.

If you want to run codes after the attaching is performed, you can use the blehook ATTACH hook.

blehook ATTACH!='<code>'

If you want to run codes when most initializations (including loading of module syntax and module complete) have ended and when ble.sh is idle (i.e., when there are no additional inputs from the user), you can use ble-import -d or ble/util/idle.push.

# You can put the codes in a file named <filename>
ble-import -d '<filename>'

# Or if you would like to directly specify a code.
ble/util/idle.push '<code>'

# Note: In bash <= 3.2, ble.sh does not support ble/util/idle.push.
# To run the code only when `ble/util/idle.push` exists (bash >= 4.0),
# you can use `ble/function#try`.
ble/function#try ble/util/idle.push '<code>'

However, the loading of the command history will be even delayed. To run the code even after the loading of the command history, you need to run the above ble-import -d or ble/util/idle.push in the ATTACH hook.

blehook ATTACH!='ble/util/idle.push "<code>"'

The calibration of ble.sh's sleep function ble/util/msleep would be run even after that in the background. To run the code in the background, you can use ble/util/idle.push-background instead of ble/util/idle.push in the above examples.

ble/util/idle.push-background '<code>'

The background processing such as auto-complete is registered even after that. To make it sure that the code is run even after such background processings registered by modules, you need to make sure that the call of ble/util/idle.push-background would be put after loading the modules. For example,

ble/util/idle.push '
  ble-import core-syntax
  ble-import core-complete
  ble/util/idle.push-background "<code>"'

[bkerin]

Hmm. The actual small thing that bugs me is sometimes when I create a new terminal and immediately cd somewhere, that cd command for some reason ends up in multiline mode. I can just push C-j so it's not a big deal, but somehow due to some race (I have no idea what) this occasionally happens.

[akinomyoga]

You can set bleopt accept_line_threshold to -1.

This is related to the detection of pasting. When the clipboard contains text including newlines and the user mistakenly pastes the content of the clipboard, random commands can be executed. To avoid such a situation, when ble.sh considers the text is part of the pasted text, it treats the newlines as a part of the text instead of the keypress Enter. However, there is no robust way to distinguish the newline in the text from the keypress Enter, so ble.sh needs to rely on a heuristic way. When the text is sent by the terminal protocol Bracketed Paste Mode, it is always treated as a part of the pasted text. But not all the terminals support the bracketed paste mode. When the newline character is directly sent, ble.sh considers it to be a part of the pasted text if it receives the newline and many succeeding characters at the same time. accept_line_threshold controls the threshold number for the "many" succeeding characters. By setting it to -1, you can completely turn off this behavior so the newlines (outside the bracketed paste mode) are always treated as the keypress Enter.

At the initialization time, the initialization takes some time so that the user's input can stack, which is then received by ble.sh at the same time and causes the behavior you observe.

[bkerin]

Thanks, this resolves this issue nicely.