Leont
commented
3 years ago I can't reproduce this, I get the same (and correct) Raw: 9, Exit: 0, signal: 9 result for both shells (on Arch Linux; dash 0.5.11, bash 5.1.8)
Open xsawyerx opened 3 years ago
Leont
commented
3 years ago I can't reproduce this, I get the same (and correct) Raw: 9, Exit: 0, signal: 9 result for both shells (on Arch Linux; dash 0.5.11, bash 5.1.8)
atoomic
commented
3 years ago To reproduce you probably want to be sure Perl was compiled with dash as the default shell?
jkeenan
commented
3 years ago @xsawyerx , can you post enough of ./perl -Ilib -V so that we can get an idea of the platform and the config_args?
Thank you very much. Jim Keenan
tonycoz
commented
3 years ago bash appears to exec the command if it's the final command:
tony@venus:.../git/perl4$ perl ../19020.pl
lrwxrwxrwx 1 root root 4 Aug 4 15:43 /bin/sh -> bash
systemd---sshd---sshd---sshd---bash---xterm---bash---perl---pstree
tony@venus:.../git/perl4$ perl ../19020.pl
lrwxrwxrwx 1 root root 4 Aug 4 15:46 /bin/sh -> dash
systemd---sshd---sshd---sshd---bash---xterm---bash---perl---sh---pstree
tony@venus:.../git/perl4$ cat ../19020.pl
my $x = 'pstree -s $$';
system 'ls -l /bin/sh';
print `$x`;which leads to the behaviour you're seeing. If you add another command after the pstree above (like an echo) the trees match.
Possibly from this in execute_cmd.c:
/* If this is a simple command, tell execute_disk_command that it
might be able to get away without forking and simply exec.
This means things like ( sleep 10 ) will only cause one fork.
If we're timing the command or inverting its return value, however,
we cannot do this optimization. */
if ((user_subshell || user_coproc) && (tcom->type == cm_simple || tcom->type == cm_subshell) &&
((tcom->flags & CMD_TIME_PIPELINE) == 0) &&
((tcom->flags & CMD_INVERT_RETURN) == 0))
{
tcom->flags |= CMD_NO_FORK;
if (tcom->type == cm_simple)
tcom->value.Simple->flags |= CMD_NO_FORK;
}
xsawyerx
commented
3 years ago I can't reproduce this, I get the same (and correct)
Raw: 9, Exit: 0, signal: 9result for both shells (on Arch Linux; dash 0.5.11, bash 5.1.8)
I'm running Dash Version: 0.5.11+git20200708+dd9ef66+really0.5.11+git20200708+dd9ef66-5ubuntu1 (Ubuntu) and 0.5.10.2-5 (Debian) where the issue exists.
The Dash manual (even on the 0.5.11 mentioned above) includes:
The current version of dash is in the process of being changed to conform with the POSIX 1003.2 and 1003.2a specifications for the shell.
I'm not sure if it's part of that or not. The changelog seems to include notes on wait() and signal catching but I haven't investigated which are relevant. It's possible it's fixed in a patch you have, Leon, or that there's a patch that the Ubuntu distro I'm quoting above did not apply.
To reproduce you probably want to be sure Perl was compiled with dash as the default shell?
I'm not sure this is the case.
Perl details on one on one of the Ubuntu systems:
Perl details on one of the Debian systems:
jkeenan
commented
3 years ago We should consider the possibility that these results may be Linux-specific. So far I am unable to reproduce the bash/dash difference on FreeBSD-12 (though I concede I have never tried varying targetsh before).
$ uname -mrs
FreeBSD 12.2-STABLE amd64
[perlmonger: jkeenan] $ which sh
/bin/sh
[perlmonger: jkeenan] $ perl -e'`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9That's with perl-5.32.0 (the "vendor perl"). I get the same result at blead:
[perlmonger: perl] $ gitcurr
blead
[perlmonger: perl] $ git describe
v5.35.2-111-g972308c272
[perlmonger: perl] $ regular_configure
...
[perlmonger: perl] $ grep '^targetsh' config.sh
targetsh='/bin/sh'
[perlmonger: perl] $ make test_prep
...
[perlmonger: perl] $ ./perl -Ilib -e'`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
[perlmonger: perl] $ git clean -dfxTargeting bash
[perlmonger: perl] $ $ sh ./Configure -des -Dusedevel -Dtargetsh=/usr/local/bin/bash -Duseithreads -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
...
[perlmonger: perl] $ grep '^targetsh' config.sh
targetsh='/usr/local/bin/bash'
[perlmonger: perl] $ make test_prep
...
[perlmonger: perl] $ bash
[jkeenan@perlmonger /usr/home/jkeenan/gitwork/perl]$ ./perl -Ilib -e'`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
[jkeenan@perlmonger /usr/home/jkeenan/gitwork/perl]$ git clean -dfx; exitTargeting dash
[perlmonger: perl] $ sh ./Configure -des -Dusedevel -Dtargetsh=/usr/local/bin/dash -Duseithreads -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
...
[perlmonger: perl] $ grep '^targetsh' config.sh
targetsh='/usr/local/bin/dash'
[perlmonger: perl] $ make test_prep
...
[perlmonger: perl] $ dash
[perlmonger: \W] $ ./perl -Ilib -e'`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
[perlmonger: \W] $ git clean -dfx; exit
toddr
commented
3 years ago I think it's safe to blame this as a bug on dash 0.5.10. Version 0.5.11 seems to not have this undesirable behavior. How do we feel about adding a test to codify what we expect the behavior to be for this?
xsawyerx
commented
3 years ago I think it's safe to blame this as a bug on dash 0.5.10. Version 0.5.11 seems to not have this undesirable behavior. How do we feel about adding a test to codify what we expect the behavior to be for this?
One of my cases is version 0.5.11, though. Maybe it's a certain patch?
I think adding a test is a good idea.
toddr
commented
2 years ago The pull request for the test now shows that even the smokers on github are broken. https://github.com/Perl/perl5/pull/19152
CC @tonycoz
wolfsage
commented
2 years ago When we execute a subshell because of metacharacters, and that subshell sticks around, it is what detects that the child perl has died with signal 9, and returns 137 which perl thinks is its exit code and shifts it over 8 to 35072.
This isn't really specific to dash - the difference between dash and bash were pointed out by tonyc above, that is: bash is smart and instead of fork/execing a child process, it just replaces itself with the intended process if it can.
But if it can't, you get the same behaviour:
my $res = system qq($^X -e'\''kill "KILL", \$\$; sleep 100'\'' && sleep 1);The && sleep 1 forces bash to stick around, and then you get the same problem.
toddr
commented
2 years ago My concern here is that based on the designated shell perl is compiled on, Perl programs which use system/qx will unexpectedly end up with different values for $?. It seems like this is something we should protect the perl coder from. Perl should provide a reliable behavior for $?.
@wolfsage are you suggesting the test in the pull request is an invalid test?
Leont
commented
2 years ago My concern here is that based on the designated shell perl is compiled on, Perl programs which use system/qx will unexpectedly end up with different values for $?. It seems like this is something we should protect the perl coder from. Perl should provide a reliable behavior for $?.
You can protect yourself from that by using the list form instead of the single-argument form of system. The latter will use the local sh, and we are not responsible for sh's behavior.
wolfsage
commented
2 years ago I think $? can/should be reliable here, and the fact that it did, or did not spawn a subshell, should not change how the return value looks. But I don't know if this is realistic.
Leont
commented
2 years ago But I don't know if this is realistic.
I think it would require writing a shell of our own.
xsawyerx
commented
2 years ago I misunderstood Tony's explanation for it. Thank you for clarifying, @wolfsage. And thanks, @tonycoz, for researching and explaining.
It seems to me that what we're saying is: There's a bug in an older version of a shell that renders $? inaccurate at times.
We can do several things:
perlvar in case someone hits it again. (I'm happy to provide the patch.)I should note that this came up not from fuzzing (which could explain a "non-feasible sequence of characters or action") but from actual code that included something along those lines and flaked out on certain boxes. That is, this actually affected a user.
tonycoz
commented
2 years ago I've done some testing with various versions of dash, and looked into what patches Debian and Ubuntu apply.
The tests:
0.5.11.4:
tony@venus:.../git/perl5$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
tony@venus:.../git/perl5$ grep targetsh config.sh
config_args='-des -Dusedevel -Dtargetsh=/home/tony/local/dash-0.5.11.4/bin/dash'
config_arg3='-Dtargetsh=/home/tony/local/dash-0.5.11.4/bin/dash'
targetsh='/home/tony/local/dash-0.5.11.4/bin/dash'
0.5.10.2:
tony@venus:.../git/perl5$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
tony@venus:.../git/perl5$ grep targetsh config.sh
config_args='-des -Dusedevel -Dtargetsh=/home/tony/local/dash-0.5.10.2/bin/dash'
config_arg3='-Dtargetsh=/home/tony/local/dash-0.5.10.2/bin/dash'
targetsh='/home/tony/local/dash-0.5.10.2/bin/dash'
0.5.10(.0)
tony@venus:.../git/perl5$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Raw: 9, Exit: 0, signal: 9
tony@venus:.../git/perl5$ grep targetsh config.sh
config_args='-des -Dusedevel -Dtargetsh=/home/tony/local/dash-0.5.10/bin/dash'
config_arg3='-Dtargetsh=/home/tony/local/dash-0.5.10/bin/dash'
targetsh='/home/tony/local/dash-0.5.10/bin/dash'
Debian (oldstable)
tony@venus:.../git/perl5$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Killed
Raw: 35072, Exit: 137, signal: 0
tony@venus:.../git/perl5$ grep targetsh config.sh
config_args='-des -Dusedevel -Dtargetsh=/bin/dash'
config_arg3='-Dtargetsh=/bin/dash'
targetsh='/bin/dash'
Ubuntu 21.04
tony@ubuntu21-04:~/dev/perl/git/perl$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Killed
Raw: 35072, Exit: 137, signal: 0
tony@ubuntu21-04:~/dev/perl/git/perl$ grep targetsh config.sh
targetsh='/bin/sh'
tony@ubuntu21-04:~/dev/perl/git/perl$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Sep 30 10:36 /bin/sh -> dashNote that the unpatched dash builds all produce the desired behaviour.
From looking at the Debian patch tracker, all of sid (bleeding edge), bookworm (testing), bullseye (stable), buster (oldstable) include this patch:
src/main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main.c b/src/main.c
index b2712cb..12dfc6e 100644
--- a/src/main.c
+++ b/src/main.c
@@ -169,7 +169,7 @@ state2:
state3:
state = 4;
if (minusc)
- evalstring(minusc, sflag ? 0 : EV_EXIT);
+ evalstring(minusc, 0);
if (sflag || minusc == NULL) {
state4: /* XXX ??? - why isn't this before the "if" statement */Ubuntu also apply the same patch, per https://git.launchpad.net/ubuntu/+source/dash/tree/debian/patches/0004-SHELL-Disable-sh-c-command-sh-c-exec-command-optimiza.diff
If I apply that to the original sources, I get the same unexpected behaviour:
0.5.11.4 + https://sources.debian.org/patches/dash/0.5.11+git20210120+802ebd4-1/0004-SHELL-Disable-sh-c-command-sh-c-exec-command-optimiza.diff/
tony@venus:.../git/perl5$ ./perl -e '`$^X -e'\''kill "KILL", \$\$'\''`; printf "Raw: $?, Exit: %s, signal: %s\n", ( $? >> 8 ), ( $? & 127 );'
Killed
Raw: 35072, Exit: 137, signal: 0
You have new mail in /var/mail/tony
tony@venus:.../git/perl5$ grep targetsh config.sh
config_args='-des -Dusedevel -Dtargetsh=/home/tony/local/dash-0.5.11.4-debian/bin/dash'
config_arg3='-Dtargetsh=/home/tony/local/dash-0.5.11.4-debian/bin/dash'
targetsh='/home/tony/local/dash-0.5.11.4-debian/bin/dash'So it looks like a Debian bug (which Ubuntu inherited).[1]
The patch appears to have been added to work around a build failure with ocamlbuild Debian ocamlbuild
[1] Assuming it's a bug, I'd expect $$ to evaluate to the shell itself rather than the child used to do the kill, but I'm not sure POSIX requires that.
Description
Under
dash, (certain?) signals in calls totargetshproduce different values for$?.Steps to Reproduce
Long-form
Perl's backticks and
system()call out to shell (specifically to targetsh[1]) when they receive an argument with an escaped variable ("\$"). (Withsystem(), this only happens when there's only one argument.)On Debian,
dashis the default non-interactive shell[2] and on Ubuntu as well[3] and that's what/bin/sh(the targetsh) will point to.Perl's
$?variable is described as the status returned "by [...] backtick command, [...], or from thesystem()operator."[4] - There's an exec, await(), and then$?reflects that result.It seems like the behavior of signal catching is not consistent between shells[5] causing this lack of consistency when:
targetshis set to/bin/sh(the default).system()./bin/shis pointing todash.waiton thatsh -ccall that Perl makes is cut off by a signal.In the same paragraph I quote above on
$?, it also states that$?is a 16-bit word: "This is just the 16-bit status word returned by the traditional Unix wait() system call (or else is made up to look like it)."[4]Notably, when Perl does not receive a 16-bit word, it is "made up to look like it." I'm not sure if this means Perl did not convert whatever
dashreturned (so it could be shifted) or not, but I thought it might be relevant that Perl provides a certain expectation, but the results are different between shells.Suggestions to fix
dash.perlport.perlvarunder$?.perlipc.I don't know if suggestion 1 is correct or not (I'll leave that to the experts) but I think at least suggestions 2 to 4 should be done. I'm happy to provide the patches, if that's desired.
[1] targetsh can be configured using
-Dtargetsh=.... It is/bin/shby default. [2] https://wiki.debian.org/Shell [3] "In Ubuntu 6.10, the default system shell, /bin/sh, was changed to dash" -- https://wiki.ubuntu.com/DashAsBinSh [4] https://perldoc.perl.org/perlvar#Error-Variables [5] https://unix.stackexchange.com/questions/240723/exit-trap-in-dash-vs-ksh-and-bash