jacoco multimodule question.

[ptahchiev]

Hi there,

I'm trying to use your plugin, and from what I read on the site:

You can use JaCoCo in a multi-module project so that all modules run JaCoCo separately and let the plugin aggregate the report

well I tried this one. I have a multi-module project setup like this:

parent (no tests, no coverage)
    |---- module1 (tests and coverage reports)
    |---- module2 (no tests no coverage reports)
    |---- module 3 (tests and coverage reports)

So what I do is I run maven from the parent project to run all my unit and integration tests and some of the modules produce results. Then after build is complete and jacoco has produced reports in some of the modules, I run maven again like this:

mvn coveralls:report

again from the parent project. However the build fails with the following error:

[ERROR] Failed to execute goal org.eluder.coveralls:coveralls-maven-plugin:3.0.1:report (default-cli) on project platform: I/O operation failed: No coverage report files found -> [Help 1]

This happens because the parent has no tests and no coverage reports. How can I can have the coveralls-maven-plugin aggregate all the reports from my modules and send it to coveralls?

Thank you

[trautonen]

I think others have reported similar issues too. There might be something I haven't figured right when applying aggregation of multi module projects with a coverage tool that actually does not support aggregation (like JaCoCo). I'll dig this up and figure a solution.

[trautonen]

Tried to play around with the sample project I have here and was unable to reproduce the issue. Can you compare your config to the sample project in sample directory. It has jacoco and coveralls plugins defined in the root module. Submodules are only jars, without any additional plugin definitions.

I tried removing tests from the module2 of the sample project so the case was:

parent (no tests, no coverage)
    |---- module1 (tests and coverage reports)
    |---- module2 (no tests no coverage reports)

I ran mvn -Pjacoco clean test jacoco:report in the root project to create the jacoco reports, and then mvn -DdryRun=true coveralls:report to create the coveralls report. In this case the jacoco report was correctly found from module1 and it was used to create the coveralls report.

Have you customized the jacoco report file destination or can you figure what differs from the sample project configuration that prevents the plugin to find the coverage reports? What maven and jacoco versions you are using?

[ptahchiev]

Sorry, I forgot to mention - yes I have modified the jacoco report file destination, because I'm generating reports for both unit and integration tests. My maven plugins looks like this:

            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>${maven.jacoco.plugin.version}</version>
                <executions>
                    <execution>
                        <id>pre-unit-test</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                            <propertyName>surefireArgLine</propertyName>
                        </configuration>
                    </execution>
                    <execution>
                        <id>post-unit-test</id>
                        <phase>test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
                        </configuration>
                    </execution>
                    <execution>
                        <id>pre-integration-test</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                            <propertyName>failsafeArgLine</propertyName>
                        </configuration>
                    </execution>
                    <execution>
                        <id>post-integration-test</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.eluder.coveralls</groupId>
                <artifactId>coveralls-maven-plugin</artifactId>
                <version>${maven.coveralls.plugin.version}</version>
                <configuration>
                    <repoToken>XXXXX</repoToken>
                    <jacocoReports>
                        <jacocoReport>${project.reporting.outputDirectory}/jacoco-it/jacoco.xml</jacocoReport>
                        <jacocoReport>${project.reporting.outputDirectory}/jacoco-ut/jacoco.xml</jacocoReport>
                    </jacocoReports>
                </configuration>
            </plugin>

[trautonen]

Ok, I see now. I think the issue is that coveralls plugin is an aggregate mojo and jacoco isn't. That causes an issue that coveralls and jacoco report file paths are not actually referring to same files. Good thing is that you can get this working, bad thing is that you need quite a lot of "hard coded" paths. So when you define jacocoReports configuration option, it is only visible to the root project and not to any submodule. Due to that fact all the paths must be relative to the root project so you actually need a custom path for every module and every jacoco report file.

<jacocoReports>
  <jacocoReport>module1/target/site/jacoco-it/jacoco.xml</jacocoReport>
  <jacocoReport>module1/target/site/jacoco-ut/jacoco.xml</jacocoReport>
  ...
</jacocoReports>

This is far from optional solution, but I'm not sure if I can make this work out any reasonable way. All the paths are already expanded to full paths at runtime so I cannot even access the user defined variables to figure out that the user maybe wanted to use a module relative path. For cobertura I think this is not an issue because it can run in aggregate mode too and collect a report to the root project.

[ptahchiev]

I guess I'm stuck - I cannot use your solution, because I have more than 40 modules in my build, and I cannot use cobertura, because for some reason it breaks my build (plus it doesn't support java8). There's a jacoco issue, that is still open for aggregate test. https://github.com/jacoco/jacoco/pull/97

In the meantime, let me know if you figure out how to do it.

Thanks for your support.

[trautonen]

I've played around a bit and I guess I have a reasonable solution. I can introduce a new configuration property relativeReportDirs where you can add module relative directories where to look the coverage report files. In your case it would be /jacoco-it and /jacoco-ut. The not so good thing is that I might shoot in the leg if there are new coverage tools introduced which report file names collide. But let's figure out a new solution then. And also the relative dirs rely on some magic dirs, which are the module's build and reporting output dirs, to look up the coverage reports.

@paranoiabla do you think this is a reasonable solution and can you see any problems with it in your config or any other config? I can push a snapshot to sonatype's repo soon so you can check how it plays out.

[ptahchiev]

Sounds great. From what I understand I will have to declare it only once in my parent pom.xml which is awesome. Looking forward to your snapshot so I can test :)

[trautonen]

Ok, there. Just add sonatype's snapshot repository to your pom if you already don't have and use version 3.1.0-SNAPSHOT.

        <repository>
            <id>sonatype-nexus-snapshots</id>
            <name>Sonatype Nexus Snapshots</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>

Then add the following configuration properties to yout coveralls maven plugin configuration section:

          <relativeReportDirs>
            <relativeReportDir>/jacoco-it</relativeReportDir>
            <relativeReportDir>/jacoco-ut</relativeReportDir>
          </relativeReportDirs>

[ptahchiev]

Hmm... I don't quite get it.. When I run mvn coveralls:report from the parent project this is what I get:

[INFO] 
[INFO] --- coveralls-maven-plugin:3.1.0-SNAPSHOT:report (default-cli) @ platform ---
[INFO] Starting Coveralls job
[INFO] Using repository token <secret>
[INFO] Git commit 2ef9097 in master
[INFO] Writing Coveralls data to /home/petar/workspace/nemesis-platform/target/coveralls.json...
[INFO] Successfully wrote Coveralls data in 36ms
[INFO] Gathered code coverage metrics for 0 source files with 0 lines of code:
[INFO] - 0 relevant lines
[INFO] - 0 covered lines
[INFO] - 0 missed lines
[INFO] Submitting Coveralls data to API
[INFO] Successfully submitted Coveralls data in 2227ms for Job #72.1
[INFO] https://coveralls.io/jobs/4355872
[INFO] *** It might take hours for Coveralls to update the actual coverage numbers for a job
[INFO]     If you see question marks in the report, please be patient
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:

It generates report for the parent and sends it successfully to coveralls. But the jacoco reports are in the submodules. Someone (either jacoco plugin or coveralls plugin) needs to go reactively through the modules and gather the reports in the parent, and then send the parent to the coveralls, right?

[trautonen]

Yes, coveralls plugin goes through all the reactor maven projects (submodules) defined in your root project (or at least should). Now we are getting some weird issue, because the coveralls plugin should fail if there are no reports found... The correct output should look something like:

[INFO] Git commit aa395ba in master
[INFO] Writing Coveralls data to /coveralls-maven-plugin/sample/target/coveralls.json...
[INFO] Processing coverage report from /coveralls-maven-plugin/sample/module1/target/site/jacoco.xml
[INFO] Processing coverage report from /coveralls-maven-plugin/sample/module2/target/site/jacoco.xml
[INFO] Successfully wrote Coveralls data in 101ms

[ptahchiev]

Hmm.. Why should the coveralls plugin fail if there are no reports found? Is there any way to stop this? Some of my modules have no source nor test, so there will be no reports generated.(The idea behind those empty modules is that it's just an empty folder to remind me to implement it later)

[ptahchiev]

Sorry, my mistake. I had an old version of the 3.1.0-SNAPSHOT in my local repository. It works fine now.

[ptahchiev]

Hi there, I have jacoco question, maybe you can help me. I have a unit test for one of my classes and jacoco is generating 40% coverage on it. However then I start my integration tests and for that same class jacoco is generating 0% coverage because I have no integration test. Then the reports are submitted to coveralls - first unit tests and then integration tests, and on the coveralls website this class appears to have 0% coverage (the second report overrides the first one). Do you know how I can actually merge both reports?

[trautonen]

Short answer: You can't. Long answer: This behavior is expected because jacoco does not support real aggregation and my plugin just picks whatever coverage report of a file it bumps in first, it does not do real aggregation from different coverage reports. Maybe I could do stateful source aggregation so that I keep track of the covered sources but this would require quite a lot of rewriting the plugin and also kill the streaming it has currently. For large code bases it would be quite hard to manage this with reasonable memory consumption.

So at the moment, not supported. Though if you have good ideas how to implement this or a lot of free time, a pull request does the job. :)

[ptahchiev]

Thanks, I think the best is to open an issue for jacoco.

[trautonen]

Yea, I think this is not the simplest problem to solve as you have probably looked the JaCoCo issue and discussion in it. Let's hope they come up with a solution or cobertura and it's maven counterpart gets a development boost so we can actually start using it.

[trautonen]

There seems to be a merge goal in the jacoco plugin and I guess it could be used to solve your problem. Run your tests and merge the results in a single exec file in the root project and create a coverage report from that. I haven't used the merge goal myself so can't provide much help other than google for that.

After that you could run coveralls plugin against the merged report. Though there's a catch, the current source loaders do not work over different modules, but luckily just got a pull request that will solve the problem. Just need the time to go through the changes in it and merge.

[ptahchiev]

Hi there,

I'm just stopping here to say that it works perfectly fine with the 3.1.0-SNAPSHOT release, but when I update to the 3.1.0 release my build is broken:

[ERROR] Failed to execute goal org.eluder.coveralls:coveralls-maven-plugin:3.1.0:report (default-cli) on project platform: I/O operation failed: No coverage report files found -> [Help 1]

So the parent has no coverage reports (no unit tests, no integration tests) so it fails :(

[trautonen]

@ptahchiev if you still have the issue with JaCoCo and integration tests, you should try the latest 4.0.0-SNAPSHOT version. I have set up JaCoCo unit and integration test setup in sample project and the coverage merging is now supported.

[stevendlander]

@trautonen I had a similar issue that I tracked down finally. For my project, this error occurred when a class file got moved to a different folder, yet the package name was not updated to reflect that change.

In other words, one of my developers moved a class (Memoize2.java) one directory above it in the folder hierarchy but did not update the package name. I think the coveralls plugin was looking for the source where the package name specified and could not find it.

Hope this helps someone!

[trautonen]

@stevendlander great catch. Java does not enforce to use same directory hierarchy than package name, but it is a generally accepted convention.

The plugin relies on the same convention, otherwise it would need to do expensive recursive searches from source directories. And also classes with same name but different package would cause problems.

[vongosling]

I met the same question. But have fixed it through polishing jacoco config. :-)

May be you can refer to my update, https://github.com/apache/incubator-rocketmq/commit/3e3988d612ce128e560719230988fd06aaad3012

[bbottema]

Is this only about Maven multimodules or also GIT submodules? I'm running into a similar issue with GIT submodules in a Maven multimodule project, where the plugin uses the commit of the root module to upload the coverage results of a submodule. This results in 422:

Report submission to Coveralls API failed with HTTP status 422: Unprocessable Entity (Couldn't find a repository matching this job.) -> [Help 1]

My guess is that this is because Coveralls (as well as Codacy, Code Climate) looks for that commit in the submodule, which doesn't exist. No clue how to solve this.