Capture code coverage data

[planetf1]

Currently code coverage is captured from our 'Sonar' test runs

This is done by enabling 'jacoco' in the build. This attaches an agent to the jvm used by surefire for running Unit Tests. Coverage data is captured and saved in a file which is then uploaded to sonar

We need to look at

• [ ] How is this affected by the move of our PR builds to github actions - is it still working
• [ ] could we capture code coverage in actions instead
• [ ] how are results published and made accessible
• [ ] Are we sure FVT coverage data is captured. From the client (failsafe)? How about the server (launched from groovy)
• [ ] Ensuring developer guidelines updated with information on finding and collecting code coverage data

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

I have added an example of capturing code coverage data from gradle at : https://github.com/odpi/egeria/pull/5159

The challenge with a multi module project is aggregating the results. In maven this is particularly difficult - and would almost certainly be easier to do via a shell script post-build (in maven we would need to create a separate reporting project, explicitly add each module as a dependency, then use the jacoco reporting plugin with the aggregate target on this reporting project -- whilst in gradle that same approach can be directly in the code)

Gradle is more flexible, and where we will likely focus future build improvements, so an initial implementation is added there

Still need to look at

• [ ] setting up a merge gradle build
• [ ] integration with external tools like codecov (upload a file to ? )
• [ ] where else could we publish the results to
• [ ] whether any other in-build processing could create a better build format
• [ ] capturing data from FVTs
• [ ] what we can/should do in maven

[planetf1]

Recently I tried to add capture of code coverage data in FVTs (maven only) (above). This involved changing the configuration of the agent prep & report to surround the integration test phase, as opposed to unit tests.

However although I can see the java agent being prepared

[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ asset-consumer-fvt ---
[INFO] jacoco.argLine set to -javaagent:/Users/jonesn/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/Users/jonesn/IdeaProjects/egeria
.maven/open-metadata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/target/jacoco.exec

And then I can see that the exec plugin that is used to launch the server chassis is specifying the agent ie:

[INFO] --- process-exec-maven-plugin:0.9:start (chassis-start) @ asset-consumer-fvt ---
[INFO] arg: java
[INFO] arg: -javaagent:/Users/jonesn/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/Users/jonesn/IdeaProjects/egeria.maven/open-metad
ata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/target/jacoco.exec
[INFO] arg: -Dserver.port=10443
[INFO] arg: -Dloader.path=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/server/lib
[INFO] arg: -Dspringdoc.api-docs.enabled=false
[INFO] arg: -Djavax.net.ssl.trustStore=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/trus
tstore.p12
[INFO] arg: -Dserver.ssl.trust-store=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/trusts
tore.p12
[INFO] arg: -jar
[INFO] arg: ../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/server/server-chassis-spring-2.
11-SNAPSHOT.jar
[INFO] Full command line: java -javaagent:/Users/jonesn/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=/Users/jonesn/IdeaProjects/eger
ia.maven/open-metadata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/target/jacoco.exec -Dserver.port=10443 -Dloader.path=../../../../../../open-metadata-distributi
on/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/server/lib -Dspringdoc.api-docs.enabled=false -Djavax.net.ssl.trustStore=../../../../
../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/truststore.p12 -Dserver.ssl.trust-store=../../../../../
../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/truststore.p12 -jar ../../../../../../open-metadata-distri
bution/open-metadata-assemblies/target/egeria-2.11-SNAPSHOT-distribution/egeria-omag-2.11-SNAPSHOT/server/server-chassis-spring-2.11-SNAPSHOT.jar
[INFO] Starting process: chassis-start
[INFO] Using working directory for this process: /Users/jonesn/IdeaProjects/egeria.maven/open-metadata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/target/classes
[ERROR] Picked up JAVA_TOOL_OPTIONS: --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED
[INFO]  Project Egeria - Open Metadata and Governance

The tests then run:

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 s - in org.odpi.openmetadata.accessservices.assetconsumer.fvt.execution.AssetConsumerOMASInvalidParamet
erIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0

And then the agent reports:

[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ asset-consumer-fvt ---
[INFO] Loading execution data file /Users/jonesn/IdeaProjects/egeria.maven/open-metadata-test/open-metadata-fvt/access-services-fvt/asset-consumer-fvt/target/jacoco.exec
[INFO] Analyzed bundle 'Asset Consumer OMAS FVT' with 5 classes
[INFO]

What we see is only 5 classes reported. That makes no sense, and on inspection we see in the csv file (this is generated by the report phase, alongside the .xml file which sonar uses):

$ cat ./access-services-fvt/asset-consumer-fvt/target/site/jacoco/jacoco.csv                                                               [9:15:57]
GROUP,PACKAGE,CLASS,INSTRUCTION_MISSED,INSTRUCTION_COVERED,BRANCH_MISSED,BRANCH_COVERED,LINE_MISSED,LINE_COVERED,COMPLEXITY_MISSED,COMPLEXITY_COVERED,METHOD_MISSED,METHOD_COVERED
Asset Consumer OMAS FVT,org.odpi.openmetadata.accessservices.assetconsumer.fvt.setup,AssetOwnerFactory,208,0,6,0,58,0,8,0,5,0
Asset Consumer OMAS FVT,org.odpi.openmetadata.accessservices.assetconsumer.fvt,AssetConsumerOMASFVTSuite,40,0,6,0,16,0,5,0,2,0
Asset Consumer OMAS FVT,org.odpi.openmetadata.accessservices.assetconsumer.fvt.clientconstructors,ClientConstructorTest,93,0,0,0,24,0,4,0,4,0
Asset Consumer OMAS FVT,org.odpi.openmetadata.accessservices.assetconsumer.fvt.errorhandling,InvalidParameterTest,150,0,0,0,50,0,7,0,7,0
Asset Consumer OMAS FVT,org.odpi.openmetadata.accessservices.assetconsumer.fvt.tagging,CreateTagTest,710,0,52,0,118,0,31,0,5,0

These look like the test driver tests, not the chassis. This is despite a) The prepare agent only being done in the fvt phase, not ut. b) The surefire plugin is not in use here

The problem here most likely is either

• Since prepare agent is really only generating a cmd line, once it's run once, anywhere in the project, we'll probably get the same result each time
• could it be that the file is being ovewritten due to test data being written by the failsafe plugin (not just sirefire)

• but currently it's not clear what else should be using it

  The possible approaches to fix might include

• Use a unique filename for FVTs vs UTs - this makes the output much clearer
• Ensuring test output is always appended to the file rather than replaced (in case of multiple use use?)
• Use a unique, non-default variable name for all surefire tests (to ensure the agent configuration doesn't automatically leak into surefire without it being explicitly set) .....

[planetf1]

I have made a change to ensure

• unique filenames for UT vs FVT
  • unique argline variable for surefire vs the server chassis

However the problem remains. (see my branch 'issue3969b')

[planetf1]

See PR

Unable to get valid jacoco reporting (which is needed for sonar) even though jacoco binary file appears correct More detailed analysis and test cases needed in case of any stack/plugin bugs, but inclined to suspend further work on the maven plugin and try with gradle instead (since we are moving that way). Will leave open for now.

[planetf1]

Attempt using maven attached. However this fails with maven exec plugin issues.

At this point I suggest we stick with gradle. Will add patch & then close the associated PR for now

Patch: https://gist.github.com/f39969518c00a0fc1bc9fc243ada4230

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[cmgrote]

Looks like we already have an issue open on capturing code coverage data -- my super-quick scan suggests this still seems focused exclusively on jacoco and SonarCloud (?) Should we reach out to LF and see if there are other "standard" tools in the kit bag that we might leverage here (that might give more consistent / integrated outputs in GitHub directly)?

[planetf1]

Yes - it's a good question. codecov seems most popular on github, but is paid-for (though some capabilities free) - so good question for LF.

There is a ci/cd workgroup, though it's been rather quiet. I'll start by asking there.

[planetf1]

We should probably target the gradle build for fixing this, on the basis that is where we are moving to