How to generate executable coverage reports with stack

[rlpowell]

So my code is arranged as four executables (i.e. most/all of the code is in "module Main"/main), with an insignificantly small bit of shared library. I would very much like to see coverage information on them, and this turns out to be Hard (tm). It is worth noting that if you want to do this without fighting GHC and stack every step of the way, the solution is probably to make your executables be teeny tiny stubs, and move everything into library modules, and I'm probably going to try that next, but having made this work I thought it was worth sharing, and some of it might be appropriate to roll into stack.

I have managed to hack together a script that does this, in somewhat grotesque fashion, but everything in here is stuff that stack probably could do without too much trouble? The ghc options in particular.

https://github.com/rlpowell/hblog/commit/b3fb0173cbb4d2a4b1fcdee66019baee5595c769

And you can see the output at:

http://users.teddyb.org/~rlpowell/media/public/hblog-coverage-2017-08-02/

The key bits here:

1. Every executable gets its own .mix directory; this is accomplished by having bits like:

ghc-options: -threaded -Wall -fhpc -hpcdir hpc/tiki_to_md/

This is super important because from GHC's POV, all of these are running the same module, and they all generate Main.mix (and there doesn't seem to be any way to have a program's main module be named anything but Main), so they all collide horribly unless so separated.

2. I have to add a space to the end of each file to make sure they recompile with said coverage options -_-; see #3306

3. Because my test suite runs the various executables in various places and a various number of times, I have to collate them using "stack exec hpc sum", and in the case of the test suite itself I have to rename its tix file from hblog-test.tix to test.tix (I have no idea why).

4. The actual coverage graphs are then generate as several calls like:

   stack exec hpc -- markup --hpcdir=hpc/$name/ --destdir=cov/$name/ --verbosity=2 hpc/$name.tix

5. There are several bits here that are specific to my environment. In particular, ignore all the rsyncs, chmods, and zipping/unzipping.

[decentral1se]

Nice digging there @rlpowell.

It does appear that this process could be improved. Maybe a good next step would be to create a trivial example with several executables (using stack new) so it is easier to see exactly what is happening here.

In general, could you clear up what you think Stack might do better here? I'm not really clued in as to where the Stack coverage functionality is at but I thought this was already possible (to easily get coverage from an executable).

[rlpowell]

It's (probably?) possible to get coverage from one executable, but since the module for all executables is "Main", and they all go in the same place, whichever test runs last smashes the others.

The main thing that's needed is for stack to set the -hpcdir differently for each component, and to run hpc markup appropriately.

Having said all that, the results are really much better when the code is all moved into libraries, as then they can all be seen in one place; hpc can't deal with multiple executables for the same reason that stack is having trouble (it expects module names to be unique).

I'll see about a minimal example.

[mgsloan]

I think one reason I didn't do this is that it makes it impossible to merge coverage results from multiple executables. It would be good to support this in the case that the user doesn't care about merging coverage results. PRs appreciated!

[rlpowell]

OK, very minimal examples for you:

http://users.teddyb.org/~rlpowell/media/public/stack_multi_exe_coverage.tgz treats each executable as having important data, and makes a separate coverage directory for each. This is a bit terrible.

http://users.teddyb.org/~rlpowell/media/public/stack_multi_lib_coverage.tgz assumes that, despite having multiple executables, the executables themselves are just shims around a library call, and generates a single coverage result by throwing away all the "Main" modules.

[rlpowell]

Note that with my actual code (which uses a proper test setup so that "stack test" works), I have to do similar machinations to the second case or I get:

HPC Report Generation Error

Error: The coverage report for hblog's test-suite "hblog-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack
issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.

[rlpowell]

OK, I've got a better version of the second one; it now uses a stack-based test suite, and generates the lovely

Generating coverage report for stack-test's test-suite "hblog-test"
Error: The coverage report for stack-test's test-suite "hblog-test" did not consider any code. One possible cause of this is if your test-suite builds the library code (see stack issue #1008). It may also indicate a bug in stack or the hpc program. Please report this issue if you think your coverage report should have meaningful results.
Completed 2 action(s).
Only one tix file found in /home/rlpowell/src/stack_test/.stack-work/install/x86_64-linux/lts-8.16/8.0.2/hpc/, so not generating a unified coverage report.

An index of the generated HTML coverage reports is available at /home/rlpowell/src/stack_test/.stack-work/install/x86_64-linux/lts-8.16/8.0.2/hpc/index.html

error, and the only place that index file leads to is

┌────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────┬──────────────────────────┬────────────────────────┐
│                                                                                        │    Top Level Definitions     │       Alternatives       │      Expressions       │
│                                         module                                         ├───────────┬──────────────────┼───────┬──────────────────┼───┬────────────────────┤
│                                                                                        │     %     │ covered / total  │   %   │ covered / total  │ % │  covered / total   │
├────────────────────────────────────────────────────────────────────────────────────────┼───────────┼───┬──────────────┼───────┼───┬──────────────┼───┼─────┬──────────────┤
│  module stack-test-0.1.0.0-4cVQaPbpZDH23pctkHIZWj/LibSpec                              │       100%│2/2│              │    50%│1/2│              │70%│21/30│              │
├────────────────────────────────────────────────────────────────────────────────────────┼───────────┼───┼──────────────┼───────┼───┼──────────────┼───┼─────┼──────────────┤
├────────────────────────────────────────────────────────────────────────────────────────┼───────────┼───┼──────────────┼───────┼───┼──────────────┼───┼─────┼──────────────┤
│  Program Coverage Total                                                                │       100%│2/2│              │    50%│1/2│              │70%│21/30│              │
└────────────────────────────────────────────────────────────────────────────────────────┴───────────┴───┴──────────────┴───────┴───┴──────────────┴───┴─────┴──────────────┘

which gives coverage only of the test shim library itself.

The coverage report that ./run_tests_coverage.sh generates (which uses no special tricks except carefully-crafted stack exec hpc calls and some hackery to force rebuilds because latest git is broken in this respect (see #3306)) looks like this:

┌────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────┬──────────────────────┬──────────────────────────┐
│                                                                                            │   Top Level Definitions    │     Alternatives     │       Expressions        │
│                                           module                                           ├─────────┬──────────────────┼───┬──────────────────┼─────┬────────────────────┤
│                                                                                            │    %    │ covered / total  │ % │ covered / total  │  %  │  covered / total   │
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┬──────────────┼───┼───┬──────────────┼─────┼─────┬──────────────┤
│  module stack-test-0.1.0.0-4cVQaPbpZDH23pctkHIZWj/Lib                                      │     100%│1/1│              │ - │0/0│              │ 100%│2/2  │              │
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┼──────────────┼───┼───┼──────────────┼─────┼─────┼──────────────┤
│  module stack-test-0.1.0.0-4cVQaPbpZDH23pctkHIZWj/LibST1                                   │     100%│1/1│              │ - │0/0│              │ 100%│4/4  │              │
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┼──────────────┼───┼───┼──────────────┼─────┼─────┼──────────────┤
│  module stack-test-0.1.0.0-4cVQaPbpZDH23pctkHIZWj/LibST2                                   │     100%│1/1│              │ - │0/0│              │ 100%│4/4  │              │
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┼──────────────┼───┼───┼──────────────┼─────┼─────┼──────────────┤
│  module stack-test-0.1.0.0-4cVQaPbpZDH23pctkHIZWj/LibSpec                                  │     100%│2/2│              │50%│1/2│              │  70%│21/30│              │
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┼──────────────┼───┼───┼──────────────┼─────┼─────┼──────────────┤
├────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼───┼──────────────┼───┼───┼──────────────┼─────┼─────┼──────────────┤
│  Program Coverage Total                                                                    │     100%│5/5│              │50%│1/2│              │  77%│31/40│              │
└────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴───┴──────────────┴───┴───┴──────────────┴─────┴─────┴──────────────┘

This version is at http://users.teddyb.org/~rlpowell/media/public/stack_multi_lib_coverage_with_spec.tgz