aureliojargas
commented
3 years ago After some research, I think now I understand the Broken pipe problem.
In my problematic command txt2regex --foo | head -n 3 | sed '3 s/:.*//, after reading the third line, head is done and just exits, since it doesn't need to read any extra lines. The kernel will then signal SIGPIPE to txt2regex, so it can be aware that the pipe is no more and there's no point in trying to write to it anymore. This is the default behavior, when there's no custom trap for SIGPIPE.
But the GitHub Actions runner traps SIGPIPE, ignoring it. This can be verified by running trap with no arguments, this is what shows up as output:
trap -- '' SIGPIPESo in this case, SIGPIPE is still signaled against txt2regex by the kernel when head early exits, but since the signal is trapped and ignored by the GitHub Actions runner, it will be ignored by the txt2regex process and it will try to send new lines to stdout (which is pipe-connected to head stdin -- which is now gone). Then this fails and then the error write error: Broken pipe is reported. In other words, the printf inside txt2regex tried to print to a closed file descriptor.
The best solution would be to "undo" the GitHub Actions trap and just have a working SIGPIPE. But it looks like once set like that, children cannot change it.
This excellent article (it's in Japanese, but the Google translation is decent) explains everything in details and gives some alternatives for fixes:
-
Use
env --default-signal=PIPE your-commandto undo the trap. But unfortunately--default-signalis a fresh new option forenvthat is still not available in Ubuntu-20.04, currently the most recent Linux available in GitHub Actions.Update at 2022-09-21: Now Ubuntu 22.04 is available for GitHub Actions and the
envsolution can be used 🎉 Example: https://github.com/aureliojargas/txt2regex/pull/13 -
Use a wrapper in other language to restore original SIGPIPE behavior. A Perl example was given.
-
Silence
stderrto get rid of the "broken pipe" messages:txt2regex --foo 2>/dev/null | head -n 3 | sed '3 s/:.*//. It works in this case, since the command exits with no error, but it feels wrong. Also note that this command is running from clitest, in a fresh shell environment, wherepipefailis not set. This is not the case for the commands in the GitHub Action's YAML configuration, since the defaultbashshell command is:bash --noprofile --norc -eo pipefail {0}. Note thatpipefailis active by default, so even if you hide the "broken pipe" message, the task will fail in the CI anyway. -
Avoid commands that could trigger SIGPIPE. In my case, I can't use a bare
head. I have to consume the whole pipe to avoid raising SIGPIPE. That's why using a temporary file instead of a pipe have worked for me in this PR. Now thinking about it,sedcan emulateheadand consume the whole pipe while doing it, fixing the problem:$ trap -- '' SIGPIPE $ seq 99999 | head -n 3 1 2 3 seq: write error: Broken pipe $ seq 99999 | sed '1,3!d' 1 2 3 $
Thanks Mario Domenech Goulart and Michael Ho for helping and teaching me on that.
Update at 2022-09-21: I'm testing this issue in this sandbox PR: https://github.com/aureliojargas/github-actions-sandbox/pull/4
Besides adding the GitHub Actions configuration, I've had to fix a test (see the second commit):
The previous code fails when running in GitHub Actions:
Even when changing
head -n 3tosed 3qit failed with the same error.When doing a single sed after the first pipe it worked fine once. In the next run it failed again, with the same error. I don't know what's happening, to be honest.
Now trying with a temporary file, to avoid piping the first command. Running 3x, all of them passed. I guess we have a winner.