Provide new task to merge external clover.db with unit test generated clover.db to combine instrumented jar results with unit tests results

[ricardoamador]

I have the plugin setup to run so I can get code coverage on my unit tests and also to generate an instrumented jar. Both of these work fine. What I would like to know is how can I combine the results of running my testing against the instrumented jar with that of the instrumented unit test results? Is this even possible?

Essentially what I would like to do is the following:

1. Build and collect unit tests code coverage.
2. Run separate tests against the instrumented jar.
3. Combine those results with the unit test results.

Currently I can generate both reports with some manipulation of the build/clover files but I cannot figure out a way to combine these.

Thanks.

[Alex-Vol-SV]

It depends on how you setup your instrumented JAR tests. If you are using Gradle to run them you could make it so that the instrumented JR test uses the same clover.db file as the unit tests code coverage. Then Clover would just update the DB file with the additional coverage.

A more involved approach would require that you setup the instrumented JAR tests as a subproject resulting in a new clover DB that could be merged with the main project using the cloverAggregateDatabases task.

I have not tried either of these approaches but if you can get them to work it would be a great help to the project to document your steps.

FYI, the file where clover.db is generated can be read from the clover extension. This would be something like: clover.initString

You can use that value to inject the Clover property like this -Dclover.initstring=${clover.initString} for your instrumented JAR tests causing them to use the same DB file. This assumes you are executing a JAVA process like a server which expects system properties like this.

[ricardoamador]

Hi Alex,

I think perhaps I should have included more detail but in regards to your responses, we do not use gradle to kick off our functional and integration tests. We keep those tests separate from our source for purposes of automation. The only tests we run with gradle are the unit tests. Our functional tests python, shell, some standalone java and c++. This is the reason we need the instrumented jar.

We run automated tests against the instrumented jars to get our overall code coverage. Which I can do now with this plugin. However I have tried every means to combine the reports and db files available but nothing seems to work.

Basically I am thinking this plugin does not allow what I am doing at this point but I would love of you to tell me I am wrong.. See below.

Now what I have been able to do is generate the code coverage for the Unit tests (which is trivial) and generate the code coverage for the functional tests run against the instrumented jar. I did this by copying the coverage.db-test file to a specific location that our service expects. And this seems to work just fine. I end up with 4 files in my tests run (cargo is the name of the service I am testing):

cargo.db-test
cargo.db-test818prt_jg8b8nax
cargo.db-test818prt_jg8b8nax.1 cargo.db-test.liverec

So if I build the initial code and instrumented jar I can get the unit test coverage. To get the functional test coverage I delete everything in build/clover and copy these files into that directory and run the report again. This gives me my functional coverage.

What I have tried is:

1. copying the above 4 files without cleaning out build/coverage and when I generate the report I get the unit test coverage results.
2. copying just the cargo.db-test818prt_jg8b8nax/.1 files into the build/coverage directory with the existing unit test coverage but again I only get the unit test results.
3. Copying everything but the cargo.db-test from the functional testing to build/coverage but again I only get the unit test results when the report is generated.
4. Generating the coverage results for both sets of tests and getting the resulting database files cargo.db and cargo_func.db then attempting to combine them in build/clover and running aggregateDatabases but this did not change the outcome.

I should also mention that the cargo.db-test.liverec file always has a size of 0 bytes. Not sure if this matters as I don't know what the purpose of that file is.

Is there some way to take the generated files from the unit tests, dump them into the build/clover directory and run the generateReport to get a combine picture of the overall code coverage?

[Alex-Vol]

With what you are doing the best way I know how is to directly invoke the Ant tasks in the OpenClover JAR that perform the merge databases work. Perhaps I can add such a task for you in the plugin to take an external generated database and merge it using a special invocation. We already do this merge for multiproject setups to generate a consolidated report. I believe this could be implemented with a little effort on my part to make this a generic solution for others doing similar to what you are.

[ricardoamador]

That would be great. I did resort to using the open clover jar directly but have not gotten to test out the above. I am hoping it will merge the databases in the way I think it should. This was previously done when our build scripts were in ant but there have been a lot of architectural changes since and it would not be feasible to port the old method.

It would be great if you added a task for merging the databases!

[ricardoamador]

Just to provide an update to you. I ended up getting what I needed to work by not using this plugin, which is a bummer. But I think there may be an issue with the plugin in the way it generates the .db file. For example when I instrument my code using the clover jar directly it generates a single .db file via the init string. However this plugin generates that .db only after the report has been generated. This seems backwards to me.

What I ended up having to do is to generate an instrumented jar by writing the instrumentation, compilation and jar task calling ant directly. ant.'clover-instr' and setting the target directories for the compile and the jar tasks.

Then I used the standard code on the clover site to generate the test code coverage results.

I then combined the jars with a customer task using ant.'clover-merge' to merge my databases and then another task ant.'clover-html-report' to generate the combined report.

I tried using the plugin but I could not get it to combine the reports and the databases it generated did not look right to me.

[Alex-Vol-SV]

The plugin uses the Ant tasks to generate the databases. The databases are generated by Clover as the unit tests are executed not during the reporting phase. For multi-project configuration an extra task cloverAggregateDatabases is executed to perform the clover-merge Ant task.

The Clover execution uses intermediate files named by suffixing the base name of the db file which is derived by the test task name. What I am planning to do is add a new custom task to the plugin that can be used as a database file holder for custom builds that use instrumented JAR files. That will give Gradle a way to reference the database generated by the custom tasks and allow the cloverAggregateDatabases task to do the clover-merge execution correctly.

[ricardoamador]

Sounds good. I will keep checking back for your changes.