Documentation request: how to merge coverage from multiple runs?

[CyberShadow]

I see that https://github.com/infertux/bashcov/pull/34 was merged, so, I understand that it should be possible to merge multiple coverage runs. However, I don't understand how to actually achieve this, and don't see any documentation which explains it.

Would it be possible to document the steps necessary in order to:

1. Run a few commands via bashcov (possibly, in parallel, or on different machines)
2. Merge their coverage
3. Format the merged coverage to a HTML report / upload it to Coveralls etc.?

Thanks!

[CyberShadow]

Through experimentation, I found that SimpleCov will automatically add the results to the .resultset.json file. This leaves the questions:

• How to merge two .resultset.json files?
• How to get bashcov to not generate a HTML report (for all but the last invocation)?
• There is a .resultset.json.lock file; does that mean it's safe to run two parallel bashcov instances in the same directory?

[tomeon]

@CyberShadow -- merging of coverage results is a feature of SimpleCov; please see the "Merging results" section of SimpleCov's docs for details. Information about how Bashcov generates "command names" for the purpose of properly merging coverage results across multiple runs is covered in the README.

Would it be possible to document the steps necessary in order to:

1. Run a few commands via bashcov (possibly, in parallel, or on different machines)
2. Merge their coverage
3. Format the merged coverage to a HTML report / upload it to Coveralls etc.?

Coverage results across multiple runs should be merged automatically, as long as SimpleCov.use_merging is enabled (as you seem to have discovered). Since bashcov automatically generates an HTML report at the end of each run, the last-generated report will contain coverage results from all bashcov invocations.

As for documenting how to merge multiple .resultset.json files, how to upload the generated HTML report to Coveralls, etc. -- IMHO, those things are not within Bashcov's sphere of responsibilities.

How to get bashcov to not generate a HTML report (for all but the last invocation)?

bashcov defines a SimpleCov.at_exit block that calls SimpleCov::Result#format!; at present, there is no way to override this. However, emptying out the list of SimpleCov formatters will prevent any reports from being generated (SimpleCov.formatters.clear in your .simplecov configuration file should do the trick). It's up to the user to determine what constitutes the "last invocation" of bashcov, and to define the list of SimpleCov formatters that should be called at the end of that invocation.

I'd be more than happy to review a PR introducing the ability to override/disable the SimpleCov.at_exit block defined in bashcov, if you'd like to take a crack at it.

There is a .resultset.json.lock file; does that mean it's safe to run two parallel bashcov instances in the same directory?

I can't guarantee that it is safe to run parallel bashcov instances in the same directory; however, SimpleCov only writes results to the .resultset.json file if it can obtain an exclusive lock on .resultset.json.lock -- does that constitute a satisfactory "yes" to your question?

[CyberShadow]

please see the "Merging results" section of SimpleCov's docs for details

Great! However:

• That link should be somewhere in bashcov's README, too
• It's not clear how Ruby-specific parts of SimpleCov's documentation apply to bashcov. For example:
  • The section following the text "you could do something like this" refers to Ruby file names.
  • It's not clear what the source of the PARALLEL_TEST_GROUPS etc. is. Is it a Ruby test runner thing?
• By carefully examining it, I can infer that the text seems to talk about running tests in parallel in the same directory. I think this should be explained more clearly. And, what about running things using parallel workers (containers), and then merging the .resultset.json files?

Since bashcov automatically generates an HTML report at the end of each run, the last-generated report will contain coverage results from all bashcov invocations.

In my case, I need to run one bashcov invocation per test. With many tests, this results in a lot of wasted overhead for generating partial HTML reports every time.

It's up to the user to determine what constitutes the "last invocation" of bashcov, and to define the list of SimpleCov formatters that should be called at the end of that invocation.

I think it would be great to have an example for this in bashcov's documentation. It would be helpful for everyone not familiar with Ruby or SimpleCov.

I might have a go at it after I'm done with the current task at hand.

• How to merge two .resultset.json files?

Looks like this is actually trivial: just merge the top-level JSON objects of the files (as long as the keys, i.e. command names, don't match). jq makes it easy: https://stackoverflow.com/questions/19529688/how-to-merge-2-json-file-using-jq

[tomeon]

That link should be somewhere in bashcov's README, too

There are multiple links to SimpleCov's docs in the README, including a link to the section regarding results merging (it's at the end of the "Controlling the command name" section).

It's not clear how Ruby-specific parts of SimpleCov's documentation apply to bashcov. For example:

The section following the text "you could do something like this" refers to Ruby file names. It's not clear what the source of the PARALLEL_TEST_GROUPS etc. is. Is it a Ruby test runner thing?

What this section of the SimpleCov documentation describes is how SimpleCov determines the name of "the currently running test suite", particularly (1) how SimpleCov infers the test suite name in the absence of an explictly-defined name, and (2) how to explicitly define the test suite name via SimpleCov.command_name. There is Ruby code in this section, but nothing that's intrinsically Ruby-specific (except for the obvious fact that you can't do SimpleCov.command_name = "my-test-suite" without eo ipso writing Ruby code). For instance, PARALLEL_TEST_GROUPS does indeed appear to come from a Ruby testing tool, but SimpleCov would detect and respect the PARALLEL_TEST_GROUPS environment variable whether set from code written in Ruby, JS, Perl, Haskell, Befunge, etc.

Moreoever, the Bashcov README discusses (1) how Bashcov generates (probably-)unique values for SimpleCov.command_name, and (2) how to set SimpleCov.command_name via bashcov --command-name ... or the BASHCOV_COMMAND_NAME environment variable. It also explicitly draws a connection to the discussion of unique command names and test result merging in the SimpleCov docs.

So, to recap:

1. The SimpleCov docs a. talk about how SimpleCov automatically sets a test suite command name when given no explicit name, and b. tell you how to set SimpleCov.command_name to ensure that results from multiple test runs are properly merged; and
2. the Bashcov docs a. talk about how Bashcov chooses a test suite command name when given no explicit name, and b. tell you how to set --command-name or BASHCOV_COMMAND_NAME (and therefore SimpleCov.command_name) to ensure that results from multiple test runs are properly merged.

I'm not sure what else you think the Bashcov docs ought to address on this matter; perhaps you could propose concrete revisions to the README?

By carefully examining it, I can infer that the text seems to talk about running tests in parallel in the same directory. I think this should be explained more clearly. And, what about running things using parallel workers (containers), and then merging the .resultset.json files?

Not sure whether the "the text" refers to SimpleCov's docs or Bashcov's. Whatever the case, it seems to me that test parallelization and documentation pertaining thereto are beyond Bashcov's scope of concern. Or perhaps you are saying that SimpleCov's documentation does not do a good job of explaining the meaning or purpose of "merging results" (giving the mistaken impression that it has to do with concurrently-running tests), and that Bashcov's documentation should clear this up?

In my case, I need to run one bashcov invocation per test. With many tests, this results in a lot of wasted overhead for generating partial HTML reports every time.

At the moment, the most straightforward way to ensure that results are formatted only once per test suite run is to structure your test suite to use a single "entrypoint" script (a la spec/test_app/test_suite.sh in Bashcov's own specs). If this is not an option for you, please feel free to open an issue or PR for disabling/overriding the SimpleCov.at_exit block defined in the bashcov executable, as described in my previous comment.

Also, have you tried benchmarking test suite duration with SimpleCov.formatters cleared out, versus with SimpleCov.formatters set to a non-empty list? I've tried this, and from what I saw the difference in duration is minimal.

Looks like [merging two .resultset.json files] is actually trivial: just merge the top-level JSON objects of the files (as long as the keys, i.e. command names, don't match). jq makes it easy: https://stackoverflow.com/questions/19529688/how-to-merge-2-json-file-using-jq

I'd encourage you to avoid having to merge resultset files in the first place, if that's an option (not 100% clear on what your use-case is). If not, your thought about using jq seems reasonable (I was going to suggest using SimpleCov's own code in case something more subtle than merging of top-level JSON objects is supposed to happen, but SimpleCov likes to use singleton methods defined on modules a lot, which makes it tough to specialize SimpleCov's behavior to permit merging the contents of multiple resultset files. I'm happy to share the details of the difficulties of using SimpleCov::ResultMerger to merge multiple resultset files, but trust me -- you don't want them :/).

---

But really, this is all somewhat academic. I think it would be much more productive to get a concrete idea of what you'd like to do with Bashcov and what problems you are running into. Maybe you've got a public repo with a test suite that uses Bashcov? If so, I'd be more than happy to take a look at the repo, make suggestions about how to achieve your testing coverage goals, provide debugging assistance, etc. Experience gleaned here could then be translated into changes to Bashcov's README, or maybe wiki entries or code to put in an examples/ directory. WDYT?

[CyberShadow]

Thanks for the extensive reply. I did manage to figure out most things in the end, but I'm not working on that project currently, so I haven't put together a pull request to improve the documentation yet.

There are multiple links to SimpleCov's docs in the README

Unfortunately this isn't very helpful. Try reading the README from the perspective from someone 100% unfamiliar with Ruby or SimpleCov (this is much harder than you may think).

Going from top to bottom, I can see the following issues:

• A whole bunch of links at the top, with no accompanying indication of what they are and whether they are important.
  • What's a RubyGem? Do I need a RubyGem to use bashcov?
  • Is the "Continuous Integration" link describing how to set up bashcov with CI? Oh, it's a link to the CI page for testing CI itself, even though there's a badge that links to the exact same page just above.
  • What is SimpleCov and how is it relevant?
• Below, there are some coverage example links. There examples show only the output, which is a nice demonstration, but not how to generate them. A simple working example that the user can run on their machine is often a good place to start for users getting into unfamiliar technology.
• The "SimpleCov integration" section immediately raises the question of what is SimpleCov, how is it relevant to BashCov, and what benefit there is to the user in learning about SimpleCov as far as their usage of BashCov is concerned.
• The "See SimpleCov README for more information" sentence and link are not helpful, because the vast majority of SimpleCov's README pertains Ruby projects, and in most cases it's unclear how the information therein translates to BashCov.
• The "Controlling the command name" section is the worst of them all, because it only talks about what the feature does, and nothing to tell the user when they would need to use it. They would need to get an understanding of SimpleCov's settings to make use of it.
• The "Some gory details" section will be irrelevant to the majority of users. It should probably be moved to a separate technical document, or converted into a troubleshooting section if applicable.

Also, have you tried benchmarking test suite duration with SimpleCov.formatters cleared out, versus with SimpleCov.formatters set to a non-empty list? I've tried this, and from what I saw the difference in duration is minimal.

I have Coveralls in the formatters to upload the results, not just HTML generation. Said upload will not work when running in a clean container, so there is a real need to separate modes of execution.

I'd encourage you to avoid having to merge resultset files in the first place, if that's an option (not 100% clear on what your use-case is).

If you want to run coverage in parallel on different machines (whether physical or containers), you do need a way to merge those result sets if you need a coverage report summing up all executions.

[infertux]

Unfortunately this isn't very helpful. Try reading the README from the perspective from someone 100% unfamiliar with Ruby or SimpleCov (this is much harder than you may think).

Going from top to bottom, I can see the following issues:

[...]

Thank you @CyberShadow, this is great feedback. As someone who's been involved with Ruby for many years, it's not easy to identify what may not be obvious to someone totally unfamiliar with Ruby or SimpleCov. Statements like "What's a RubyGem? Do I need a RubyGem to use bashcov?" are actually great eye-openers (Bashcov is actually a RubyGem :)). I'll try to simplify the README and make it more generic and language-agnostic.

[tomeon]

Yes, thanks @CyberShadow -- nice list of concrete areas for improvement. Since it looks like @infertux is already tackling the docs, I'll respond to:

If you want to run coverage in parallel on different machines (whether physical or containers), you do need a way to merge those result sets if you need a coverage report summing up all executions.

Even here, there may be ways to avoid having to do any merging (other than the merging that SimpleCov itself already does, even in non-parallel setups). For instance, if the containers are running on the same host, I'd suggest bind-mounting the .resultset.json file or one of its parent directories into the containers. That way, when SimpleCov locks the resultset file in one container, it should be locked across all containers, and SimpleCov should (:crossed_fingers:) be able to merge results from all containers safely.

A quick POC demonstrating locking across multiple containers:

# itsmine.rb

require 'socket'

File.open(File.expand_path('mine', __dir__), 'r+') do |f|
  sleep rand 1..3
  f.flock(File::LOCK_EX | File::LOCK_NB) or abort "#{Socket.gethostname}: But... it's mine!"
  sleep 5
end

#!/bin/sh

# itsmine.sh

set -eu

for i in 1 2 3; do
    name="itsmine${i}"
    docker run -d -v "${PWD}:/mine" --name "$name" --hostname "$name" ruby:alpine ruby /mine/itsmine.rb
done

sleep 5

for i in 1 2 3; do
    name="itsmine${i}"
    docker logs "$name"
    docker stop "$name" 1>/dev/null 2>&1 || :
    docker rm "$name" 1>/dev/null
done

$ touch mine
$ ./itsmine.sh
43f84a3f4ff297a631a62f7a1aae6ecccf310dddf22c945298b7a777575c9a4b
316bb2c0e3832f9a4722bb8d37da31d9f1babfda1133998a6665544c78420a13
244c4ebf75f41e3cee573aa1b444a66b2680fc7a05cc2b245b50c6b35007faf2
itsmine1: But... it's mine!
itsmine3: But... it's mine!

Of course, this comes with the usual "works on my machine" caveats :).

Separate machines are probably a tougher nut to crack, but if those isolated machines share a networked filesystem that is supported by Ruby's File#flock implementation, you might be able to get away with locating the .resultset.json file, or coverage directory, etc., on the network share. I have not tested this in the slightest, though, so it's purely speculative.

[tomeon]

Also --

I have Coveralls in the formatters to upload the results, not just HTML generation. Said upload will not work when running in a clean container, so there is a real need to separate modes of execution.

This might be a good place to explore using SimpleCov's profiles feature. For instance, you could add something like this to your .simplecov:

require 'coveralls'

SimpleCov.profiles.define 'coveralls' do
  SimpleCov.formatters = [Coveralls::SimpleCov::Formatter]
end

Then add a small script like this:

# upload_results.rb

require 'simplecov'

SimpleCov.profiles.load 'coveralls'
SimpleCov::ResultMerger.merged_result.format!

Or you could just inline the SimpleCov configuration:

# upload_results_v2.rb

require 'simplecov'
require 'coveralls'

SimpleCov.formatters = [Coveralls::SimpleCov::Formatter]
SimpleCov::ResultMerger.merged_result.format!

Run the script after your tests complete, and the coverage results should be uploaded to Coveralls.io.

N.B. you may need to to twiddle SimpleCov.root and/or SimpleCov.coverage_dir depending on where the script lives in relation to the .resultset.json file.

[CyberShadow]

That way, when SimpleCov locks the resultset file in one container, it should be locked across all containers, and SimpleCov should (crossed_fingers) be able to merge results from all containers safely.

Yep, the big question is whether locks will work across bind mounts. I'm not sure if Docker provides any guarantees regarding the mounting mechanism used, as far as locks go. For instance, this probably won't work with Docker on OSX, which seems to use something more FUSE-ish.

This might be a good place to explore using SimpleCov's profiles feature.

That's pretty great. I don't think I would have discovered this on my own.

What I ended up doing was using environment variables to indicate what part of the test suite is being run: https://github.com/CyberShadow/aconfmgr/blob/master/test/.simplecov