How to get JaCoCo coverage while using "mvn jqf:repro ..." command

Hi!

I wrote little test project containing the dimMe() method of com.org.exampleclass.ExampleClass class and corresponding ExampleClassFuzzTest.java fuzzing wrapper for this method.

I'm successfully fuzzing this method using jqf-maven-plugin: mvn jqf:repro -Dclass=com.org.exampleclass.ExampleClassFuzzTest -Dmethod=testFuzzDimMe

And then I succeed reproduce function arguments from input files: mvn jqf:repro -Dclass=com.org.exampleclass.ExampleClassFuzzTest -Dmethod=testFuzzDimMe -Dinput=target/fuzz-results/com.org.exampleclass.ExampleClassFuzzTest/testFuzzDimMe/corpus/ -DprintArgs

I would like to get a JaCoCo report on the coverage achieved when running the last command. How can this be achieved? I know about the -DlogCoverage option, but I need the JaCoCo format (jacoco.exec file).

I tried installing javaagent via the MAVEN_OPTS environment variable (see run.sh file). But in the html report I see empty coverage.

Also I try run mvn test -Dtest=<filter> with a @Fuzz(repro=<file_or_dir>) annotation on the JQF test driver, but maven does not treat fuzzing tests with this annotation as unit tests and does not execute them.

Is there a way and how to link in pom.xml the execution of jqf-maven-plugin with jacoco-maven-plugin? Or is there any other way to get the correct jacoco.exec file when running the mvn jqf:repro command?

Also I try run mvn test -Dtest= with a @Fuzz(repro=) annotation on the JQF test driver, but maven does not treat fuzzing tests with this annotation as unit tests and does not execute them.

This should definitely work, and is an easy way to run all test inputs in the corpus by providing a directory.

In the JQF-Zest Example repo, if I add this annotation, then I can run after fuzzing:

$ mvn test -Dtest=PatriciaTrieTest
[...]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running examples.PatriciaTrieTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
id_000000 ::= INVALID
id_000001 ::= SUCCESS
id_000002 ::= SUCCESS
id_000003 ::= SUCCESS
id_000004 ::= SUCCESS
id_000005 ::= SUCCESS
id_000006 ::= SUCCESS
id_000007 ::= SUCCESS
id_000008 ::= SUCCESS
id_000009 ::= SUCCESS
id_000010 ::= SUCCESS
id_000011 ::= SUCCESS
id_000012 ::= SUCCESS
id_000013 ::= SUCCESS
id_000014 ::= SUCCESS
id_000015 ::= SUCCESS
id_000016 ::= SUCCESS
id_000017 ::= SUCCESS
id_000018 ::= SUCCESS
id_000019 ::= SUCCESS
id_000020 ::= SUCCESS
id_000021 ::= SUCCESS
id_000022 ::= SUCCESS
id_000023 ::= SUCCESS
id_000024 ::= SUCCESS
id_000025 ::= SUCCESS
id_000026 ::= SUCCESS
id_000027 ::= SUCCESS
id_000028 ::= SUCCESS
id_000029 ::= SUCCESS
id_000030 ::= SUCCESS
id_000031 ::= SUCCESS
id_000032 ::= SUCCESS
id_000033 ::= SUCCESS
id_000034 ::= SUCCESS
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.524 s -- in examples.PatriciaTrieTest

In your example repo, I think the dependency on Jupiter might be causing a conflict with JQF's JUnit4-based runner and spoil test discovery.

A low-level workaround might be to use bin/jqf-repro which is a command-line non-Maven way of launching the JQF repro driver (use -h for help, and you can use $(scripts/classpath.sh) to get a classpath of all the JQF JARs). Then, you can provide jacoco as a JavaAgent directly via JVM_OPTS.

JQF has also has native support for creating jacoco.exec files using the -Djqf.repro.jacocoAccumulateJar option to mvn jqf:repro, but it requires having the jacocoagent.jar file on the classpath (so not a pure Maven solution, since JaCoCo doesn't publish its unwrapped agent JAR files to Maven Central).

Thank you very much for your answer!

In your example repo, I think the dependency on Jupiter might be causing a conflict with JQF's JUnit4-based runner and spoil test discovery.

Indeed, downgrade to JUnit4 solved the problem when using @fuzz(repro=) annotation.

JQF has also has native support for creating jacoco.exec files using the -Djqf.repro.jacocoAccumulateJar option to mvn jqf:repro, but it requires having the jacocoagent.jar file on the classpath

This is a very interesting option, but I couldn't get it to work.

I add this lines in pom.xml:

<dependency>
  <groupId>org.jacoco</groupId>
  <artifactId>org.jacoco.agent</artifactId>
  <classifier>runtime</classifier>
  <version>${jacoco-maven-plugin.version}</version>
</dependency>

In the command mvn dependency:tree output I get org.jacoco:org.jacoco.agent:jar:runtime:0.8.10:compile.

But when I try to run mvn jqf:repro -Dclass=com.org.exampleclass.ExampleClassFuzzTest -Dmethod=testFuzzDimMe -Dinput=target/fuzz-results/com.org.exampleclass.ExampleClassFuzzTest/testFuzzDimMe/corpus/ -DprintArgs -Djqf.repro.jacocoAccumulateJar arises java.lang.ClassNotFoundException: org.jacoco.agent.rt.RT.

What am I doing wrong?

As I mentioned earlier, adding the jacocoagent.jar file to classpath is not a pure-Maven solution, because of how JaCoCo is deployed to Maven Central.

What you have as a dependency is the org.jacoco:org.jacoco.agent package, which does not directly contain the class org.jacoco.agent.rt.RT or related classes. Instead, it includes jacocoagent.jar wrapped inside the JAR. See:

$ jar tf ~/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12.jar 
META-INF/
META-INF/MANIFEST.MF
org/
org/jacoco/
org/jacoco/agent/
META-INF/maven/
META-INF/maven/org.jacoco/
META-INF/maven/org.jacoco/org.jacoco.agent/
jacocoagent.jar
about.html
org/jacoco/agent/package-info.class
org/jacoco/agent/AgentJar.class
META-INF/maven/org.jacoco/org.jacoco.agent/pom.xml
META-INF/maven/org.jacoco/org.jacoco.agent/pom.properties

The Maven JAR they publish is a wrapper for the JAR that actually contains the classes. This is fine for their use case of using the Maven package as a javaagent, but unfortunately not useful when we want to call the agent classes programatically.

You could create a pure-Maven solution by using some other plugins that unwrap this JAR at test time and include it in your dependencies somehow. I haven't gone that deep. For research experiments where we needed JaCoCo coverage results, we simply pulled the jacocoagent.jar as a system dependency.

You could create a pure-Maven solution by using some other plugins that unwrap this JAR at test time and include it in your dependencies somehow.

I'm not very experienced in Java yet and I don't really know how to implement this. :(

For research experiments where we needed JaCoCo coverage results, we simply pulled the jacocoagent.jar as a system dependency.

Do you mean using bin/jqf-repro option in your research experiments? Or the -Djqf.repro.jacocoAccumulateJar option to mvn jqf:repro will work too in case jacocoagent.jar as a system dependency?

I unpack jacoco-0.8.10.zip into /home/user/Fuzzing/jacoco-0.8.10 directory and then I trying to run the following bash script:

export CLASSPATH="$CLASSPATH:/home/user/Fuzzing/jacoco-0.8.10/lib" # option 1 export MAVEN_OPTS="$MAVEN_OPTS -classpath /home/user/Fuzzing/jacoco-0.8.10/lib" # option 2 mvn jqf:repro -Dclass=com.org.exampleclass.ExampleClassFuzzTest -Dmethod=testFuzzDimMe -Dinput=target/fuzz-results/com.org.exampleclass.ExampleClassFuzzTest/testFuzzDimMe/corpus/ -DprintArgs -Djqf.repro.jacocoAccumulateJar

But I get ClassNotFoundException again.

Also I try correct this lines in pom.xml:

<dependency>
  <groupId>org.jacoco</groupId>
  <artifactId>org.jacoco.agent</artifactId>
  <classifier>runtime</classifier>
  <version>${jacoco-agent-version}</version>
  <scope>system</scope>
  <systemPath>${project.basedir}/../jacoco-0.8.10/lib/jacocoagent.jar</systemPath>
</dependency>

But this does not give the desired effect.

rohanpadhye / JQF

How to get JaCoCo coverage while using "mvn jqf:repro ..." command #256