Allow merging coverage data files and creating a report off of the combined coverage

[scottg489-tw]

Is your feature request related to a problem? Please describe. We have multiple servers reporting coverage to a local Redis instance. This means that looking at coverage on one server doesn't give the full picture.

Describe the solution you'd like We would like to be able to:

1. Generate a coverage data file
2. Merge it with another coverage data file
3. Generate a report off of the merged coverage data

Describe alternatives you've considered It seems like this functionality may have been implemented in #18, at least somewhat. However, we were not able to get this working.

We're on a pretty old version (1.2) so even though the example would no longer work on newer versions, we were able to implement what it appears to be doing. However, the resulting coverage report is empty. Here are the tasks we defined:

task :write_coverage_data do |task, args|
  file_name = args.to_a[0]
  data = JSON.generate Coverband::Reporter.get_current_scov_data
  File.write(file_name, data)
end
task :aggregate_coverage do |task, args|
  coverage_data_to_aggregate = []
  puts args.extras.each { |file_name|
    coverage_data_to_aggregate.append(JSON.parse(File.read(file_name)))
  }
  Coverband::Reporter.report :additional_scov_data => coverage_data_to_aggregate
end

The write_coverage_data task is meant to be run on each server to generate the coverage data file. Then all the files would be downloaded to a central location and aggregate_coverage would run and merge all these coverage data files together and generate the HTML report.

I'm not entirely sure where this implementation is failing. The coverage data files are generated and appear to have coverage data (~80K in size). So that would leave the problem with the reporting. The reporting task seems to require a local Redis instance to be running or fails.

Additional context I understand that we could attempt to point all of our servers at a central Redis instance, but this isn't feasible at the moment.

[danmayer]

hmm... so we basically dropped static report generation and mering of data files a long time ago... it adds a lot of complexity and in general my response has been per environment you want shared metrics point to a single redis... I don't think we will bring back the merging and such as it is a rare need outside of the testing environment which we don't target as much as production...

After that we had worked on a coverband service that would act as an aggregator... It has been paused for a bit, it was a pay service not free like the open source. If something like that existed would it interest your team?

[scottg489-tw]

Hey Dan! Thanks for this great tool.

So it turned out the issue was with paths. The paths on the server where I created the coverage data files obviously aren't the same as the paths locally where I'm aggregating them. It seems like there's some code that is meant to fix this, but it doesn't apply for the additional coverage files.

My fix was just to change the paths in the coverage data files so they matched my local path.

Regarding the dropped support, we may be able to centralize the coverage to a single redis server at the point where we decide to update coverband to a newer version. However, the merged data files could still be useful in situations where we wanted to keep a history of coverage. For instance, backing up coverage to files during a deploy before we clear the data. Of course the coverage data may not apply anymore if those lines of code changed, but it would still be applicable for most of the application.

It's possible we might be interested in an aggregator service, but we'd need to know more about specifics and pricing.

[danmayer]

yeah I am still sorting out what pricing would be and if I think I will have the time to fully support it... I am looking at merging another end point that someone wanted with a json dump of the data... that would get pretty close to helping support some of the use cases you have at least in terms of collecting all the data in a re-usable format... I will consider this feature a bit more, but hopefully find ways to keep it really light weight.

[danmayer]

OK, there are two rake tasks that when used together can build out an aggregated report.

The new rake tasks were added to tasks.rb and are documented in the comments around those tasks...

You can generate json reports on all y our servers and put them... somewhere... then collect them all to some system and run the other rake task to aggregate the data.

• FOR_MERGED_REPORT=true bundle exec rake coverband:coverage_json to generate the JSON files
• bundle exec rake coverband:aggregate_coverage to merge the files