Feature request: option to copy stderr of run statements to another file / file descriptor

[pawamoy]

Here is my use-case:

I wrote a shell script to compute code coverage. It uses the xtrace/debug mode of Zsh/Bash to know which lines of which files have been executed.

Initially I wrote it for Bash only, but Bash's LINENO variable is... not always useful. Zsh's LINENO is much better, so I added support for Zsh.

Now, this script will generally be used to run test suites. Typically, with Bats for Bash and ZUnit for Zsh. It works wonderfully with Bats (except for the LINENO detail), because I can use Bash's BASH_XTRACEFD variable which redirect the xtrace output to another file descriptor than stderr.

Unfortunately, such a variable does not exist in Zsh: the xtrace output is always sent to stderr. This is a problem because the xtrace otuput is then captured by Zunit (in run statements), and therefore made unavailable to my coverage script.

This is where my feature request comes to play: while waiting for Zsh to implement such a ZSH_XTRACEFD variable, or similar (I sent a mail to the zsh-workers mailing list asking for it), I would like Zunit to provide an option to copy the stderr of a run command to another file or file descriptor of my choice. Some kind of tee on stderr, something like 2> >(tee >&1 "$zunit_copy_stderr_file").

This example (which is not tested and maybe not working) is explained as:

• send stderr to a new file descriptor
• which uses tee to send it back
  • to stdout: this is what Zunit currently does: https://github.com/zunit-zsh/zunit/blob/767734875893319866b1f244556f1f4aecba1dcb/src/commands/run.zsh#L185 or the async version: https://github.com/zunit-zsh/zunit/blob/767734875893319866b1f244556f1f4aecba1dcb/src/commands/run.zsh#L182
  • and to a file whose path is contained in the zunit_copy_stderr_file variable.

This feature would allow me to copy the stderr, and therefore xtrace output of Zsh into a file of my choice to further process it later.

Would you be willing to integrate such an option in Zunit? If yes, I can try and open a PR in a few days.

P.S.: I saw earlier that you have a "coverage" branch, but that it was not updated since a relatively long time. It would be great if Zunit had integrated code coverage, but I think an independent script is still a good idea, not being tied to a particular testing framework. This is why I'm opening this ticket :slightly_smiling_face:

[pawamoy]

In fact, it can be simplified as: add an option to prevent the capture of stderr in run statements. I've implemented it here: https://github.com/pawamoy/zunit/commit/13ee641717711681d387cbe6efea6310f7b8aa1c (quick and dirty)

[molovo]

@pawamoy I like this idea, and the implementation looks good, so thanks.

I'd be very happy to accept a pull request if you submit one, but would you mind adding a couple of tests for the new option as well please?

Also, regarding coverage, I'd be very interested in seeing a project where you have this working, if and when you get one set up. I tried very hard to add a coverage feature to ZUnit, and struggled miserably, so if you have a working solution I'd like to point people towards it in the documentation.

[pawamoy]

@molovo Great! Of course I can open a PR with additional tests :slightly_smiling_face: Do you have any other remarks about the naming of the option or else before I open the PR?

If you want to take a look at my script, it's here: https://github.com/shellm-org/cover

My code is not as readable and beautiful as yours, but I will definitely improve it :sweat_smile:

A few technical notes about coverage since I think you are interested:

• coverage for Bash will never work correctly, because of how the LINENO variable behaves. See https://savannah.gnu.org/support/?func=detailitem&item_id=109686 (opening the link seems to be problematic on mobile browser Brave, the & gets corrupted or something, try to write the GET params by hand).
• fortunately, LINENO in Zsh behaves as expected. Now the difficulty is to handle multi-line commands and multi-line strings!
  • multi-line commands are in fact easy: check for groups of lines that finish with \, plus one line of context after it (grep -A 1), mark them as covered if the top one is.
  • multi-line strings are... not doable :slightly_frowning_face: I mean, unless there's a parser for Zsh somewhere, that could be used in a Zsh script haha. There is no way to parse multi-line strings correctly with regular expressions.

To conclude, I'll only get 87.50% coverage (missing line 5) on the following snippet instead of 100% (Bash only gets 75%, missing lines 1, 3-6, 8-10):

echo $(
  # comment
  echo "hello" \
    "world
    thanks" | # comment
      cat | ( # comment
        # comment
        cat -n # comment
      ) | cat && # comment
      echo bye
)

[molovo]

@pawamoy maybe just expand it to --no-capture-err so it's a little more obvious what it does 👍

I'll take a look at the coverage later this evening and get back to you

[pawamoy]

Ah, my code on GitHub is not up-to-date at all. I'll push before this evening for you to see the latest version then :)

[pawamoy]

See closed PR #102, abandoning this for a cleaner ZSH_XTRACEFD variable.