Closed aureliojargas closed 3 years ago
After some research, I think now I understand the Broken pipe
problem.
In my problematic command txt2regex --foo | head -n 3 | sed '3 s/:.*//
, after reading the third line, head
is done and just exits, since it doesn't need to read any extra lines. The kernel will then signal SIGPIPE
to txt2regex
, so it can be aware that the pipe is no more and there's no point in trying to write to it anymore. This is the default behavior, when there's no custom trap for SIGPIPE.
But the GitHub Actions runner traps SIGPIPE
, ignoring it. This can be verified by running trap
with no arguments, this is what shows up as output:
trap -- '' SIGPIPE
So in this case, SIGPIPE is still signaled against txt2regex
by the kernel when head
early exits, but since the signal is trap
ped and ignored by the GitHub Actions runner, it will be ignored by the txt2regex
process and it will try to send new lines to stdout (which is pipe-connected to head stdin -- which is now gone). Then this fails and then the error write error: Broken pipe
is reported. In other words, the printf
inside txt2regex
tried to print to a closed file descriptor.
The best solution would be to "undo" the GitHub Actions trap and just have a working SIGPIPE. But it looks like once set like that, children cannot change it.
This excellent article (it's in Japanese, but the Google translation is decent) explains everything in details and gives some alternatives for fixes:
Use env --default-signal=PIPE your-command
to undo the trap. But unfortunately --default-signal
is a fresh new option for env
that is still not available in Ubuntu-20.04, currently the most recent Linux available in GitHub Actions.
Update at 2022-09-21: Now Ubuntu 22.04 is available for GitHub Actions and the
env
solution can be used 🎉 Example: https://github.com/aureliojargas/txt2regex/pull/13
Use a wrapper in other language to restore original SIGPIPE behavior. A Perl example was given.
Silence stderr
to get rid of the "broken pipe" messages: txt2regex --foo 2>/dev/null | head -n 3 | sed '3 s/:.*//
. It works in this case, since the command exits with no error, but it feels wrong. Also note that this command is running from clitest, in a fresh shell environment, where pipefail
is not set. This is not the case for the commands in the GitHub Action's YAML configuration, since the default bash
shell command is: bash --noprofile --norc -eo pipefail {0}
. Note that pipefail
is active by default, so even if you hide the "broken pipe" message, the task will fail in the CI anyway.
Avoid commands that could trigger SIGPIPE. In my case, I can't use a bare head
. I have to consume the whole pipe to avoid raising SIGPIPE. That's why using a temporary file instead of a pipe have worked for me in this PR. Now thinking about it, sed
can emulate head
and consume the whole pipe while doing it, fixing the problem:
$ trap -- '' SIGPIPE
$ seq 99999 | head -n 3
1
2
3
seq: write error: Broken pipe
$ seq 99999 | sed '1,3!d'
1
2
3
$
Thanks Mario Domenech Goulart and Michael Ho for helping and teaching me on that.
Update at 2022-09-21: I'm testing this issue in this sandbox PR: https://github.com/aureliojargas/github-actions-sandbox/pull/4
The mentioned PR #13 upgraded the CI image to Ubuntu 22.04 so we can use the env --default-signal=PIPE
trick to fix the issue for good 🎉
Besides adding the GitHub Actions configuration, I've had to fix a test (see the second commit):
The previous code fails when running in GitHub Actions:
Even when changing
head -n 3
tosed 3q
it failed with the same error.When doing a single sed after the first pipe it worked fine once. In the next run it failed again, with the same error. I don't know what's happening, to be honest.
Now trying with a temporary file, to avoid piping the first command. Running 3x, all of them passed. I guess we have a winner.