zunit-zsh / zunit

A powerful testing framework for ZSH projects
https://zunit.xyz
MIT License
208 stars 23 forks source link

Feature request: option to copy stderr of run statements to another file / file descriptor #101

Closed pawamoy closed 5 years ago

pawamoy commented 5 years ago

Here is my use-case:

I wrote a shell script to compute code coverage. It uses the xtrace/debug mode of Zsh/Bash to know which lines of which files have been executed.

Initially I wrote it for Bash only, but Bash's LINENO variable is... not always useful. Zsh's LINENO is much better, so I added support for Zsh.

Now, this script will generally be used to run test suites. Typically, with Bats for Bash and ZUnit for Zsh. It works wonderfully with Bats (except for the LINENO detail), because I can use Bash's BASH_XTRACEFD variable which redirect the xtrace output to another file descriptor than stderr.

Unfortunately, such a variable does not exist in Zsh: the xtrace output is always sent to stderr. This is a problem because the xtrace otuput is then captured by Zunit (in run statements), and therefore made unavailable to my coverage script.

This is where my feature request comes to play: while waiting for Zsh to implement such a ZSH_XTRACEFD variable, or similar (I sent a mail to the zsh-workers mailing list asking for it), I would like Zunit to provide an option to copy the stderr of a run command to another file or file descriptor of my choice. Some kind of tee on stderr, something like 2> >(tee >&1 "$zunit_copy_stderr_file").

This example (which is not tested and maybe not working) is explained as:

This feature would allow me to copy the stderr, and therefore xtrace output of Zsh into a file of my choice to further process it later.

Would you be willing to integrate such an option in Zunit? If yes, I can try and open a PR in a few days.

P.S.: I saw earlier that you have a "coverage" branch, but that it was not updated since a relatively long time. It would be great if Zunit had integrated code coverage, but I think an independent script is still a good idea, not being tied to a particular testing framework. This is why I'm opening this ticket :slightly_smiling_face:

pawamoy commented 5 years ago

In fact, it can be simplified as: add an option to prevent the capture of stderr in run statements. I've implemented it here: https://github.com/pawamoy/zunit/commit/13ee641717711681d387cbe6efea6310f7b8aa1c (quick and dirty)

molovo commented 5 years ago

@pawamoy I like this idea, and the implementation looks good, so thanks.

I'd be very happy to accept a pull request if you submit one, but would you mind adding a couple of tests for the new option as well please?

Also, regarding coverage, I'd be very interested in seeing a project where you have this working, if and when you get one set up. I tried very hard to add a coverage feature to ZUnit, and struggled miserably, so if you have a working solution I'd like to point people towards it in the documentation.

pawamoy commented 5 years ago

@molovo Great! Of course I can open a PR with additional tests :slightly_smiling_face: Do you have any other remarks about the naming of the option or else before I open the PR?

If you want to take a look at my script, it's here: https://github.com/shellm-org/cover

My code is not as readable and beautiful as yours, but I will definitely improve it :sweat_smile:

A few technical notes about coverage since I think you are interested:

To conclude, I'll only get 87.50% coverage (missing line 5) on the following snippet instead of 100% (Bash only gets 75%, missing lines 1, 3-6, 8-10):

echo $(
  # comment
  echo "hello" \
    "world
    thanks" | # comment
      cat | ( # comment
        # comment
        cat -n # comment
      ) | cat && # comment
      echo bye
)
molovo commented 5 years ago

@pawamoy maybe just expand it to --no-capture-err so it's a little more obvious what it does 👍

I'll take a look at the coverage later this evening and get back to you

pawamoy commented 5 years ago

Ah, my code on GitHub is not up-to-date at all. I'll push before this evening for you to see the latest version then :)

pawamoy commented 5 years ago

See closed PR #102, abandoning this for a cleaner ZSH_XTRACEFD variable.