Work around missing JUnit reporting entry support in Gradle

[CodeLowSky]

I tried to get a basic test running with gradle but it seems like jazzer is trying to instrument the gradle workers and fails somewhere.

$ gradle build
INFO: Instrumented org.gradle.api.internal.tasks.testing.processors.DefaultStandardOutputRedirector$DiscardAction with custom hooks only (took 1 ms, size +0%)

> Task :app:test

DummyFuzzTest > dummyFuzz(FuzzedDataProvider) > dummy.DummyFuzzTest.initializationError FAILED
    java.lang.IllegalStateException at AgentInstaller.java:55
        Caused by: java.lang.reflect.InvocationTargetException at NativeMethodAccessorImpl.java:-2
            Caused by: java.lang.VerifyError at InstrumentationImpl.java:-2

build.gradle

plugins {
    // Apply the application plugin to add support for building a CLI application in Java.
    id 'application'
}

repositories {
    // Use Maven Central for resolving dependencies.
    mavenCentral()
}

dependencies {
    testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'

    testImplementation 'com.code-intelligence:jazzer-api:0.13.0'
    testImplementation 'com.code-intelligence:jazzer-junit:0.13.0'
}

application {
    // Define the main class for the application.
    mainClass = 'dummy.App'
}

tasks.named('test') {
    // Use JUnit Platform for unit tests.
    useJUnitPlatform()
}

App.java

package dummy;

public class App {
    public int add(int a, int b) {
        return a+b;
    }

    public static void main(String[] args) {
        System.out.println(new App().add(1, 2));
    }
}

AppTest.java

package dummy;

import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;

class DummyFuzzTest {
    @FuzzTest
    void dummyFuzz(FuzzedDataProvider data) {
        new App().add(data.consumeInt(), data.consumeInt());
    }
}

Any idea how to get this running?

[fmeum]

A complete stack trace of the Gradle would be very helpful - a VerifyError hints at something more serious.

That said, you can control the instrumentation with the jazzer.instrument JUnit configuration parameter, see https://github.com/CodeIntelligenceTesting/jazzer/blob/main/examples/junit/src/test/resources/junit-platform.properties.

[CodeLowSky]

Attached is the stack trace, thanks for looking into this. In the meanwhile I try the instrument option.

trace.txt

[CodeLowSky]

So I've played around a bit and added the junit-platform.properties.

jazzer.instrument=dummy.**

The build is working now but the test executions seems a bit off. I added byteFuzz test case from the examples which should fail but it does not.

@FuzzTest
    void byteFuzz(byte[] data) {
        if (data.length < 1) {
            return;
        }
        if (data[0] % 2 == 0) {
            fail();
        }
    }

It seems like the test cases are detected, because it shows two cases in TEST-dummy.DummyFuzzTest.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="dummy.DummyFuzzTest" tests="2" skipped="0" failures="0" errors="0" timestamp="2022-10-27T12:20:19" hostname="ISCN5CG2232JPT" time="2.364">
  <properties/>
  <testcase name="&lt;empty input&gt;" classname="dummy.DummyFuzzTest" time="2.324"/>
  <testcase name="&lt;empty input&gt;" classname="dummy.DummyFuzzTest" time="0.007"/>
  <system-out><![CDATA[INFO: Loaded 163 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
INFO: Loaded 50 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
INFO: Loaded 76 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
INFO: Instrumented dummy.DummyFuzzTest with custom hooks only (took 36 ms, size +0%)
INFO: Instrumented dummy.App with custom hooks only (took 2 ms, size +0%)
]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

Am I missing something? Also why can't the testcase names be resolved?

[fmeum]

By default, the tests are only run on a fixed set of inputs stored in the src/test/resources/dummy/DummyFuzzTestInputs directory (for your example), plus the empty input. This directory is populated with findings encountered during actual fuzzing runs, which you can start by running the tests with the environment variable JAZZER_FUZZ set to 1. I tried that on your example and got a finding emitted into the directory that I could then reproduce when running the test without JAZZER_FUZZ set.

We actually use JUnit's reporting entry feature to tell you about all this right in your test logs, which works well with Maven but for some reason doesn't show up with Gradle.

[fmeum]

Turns out https://github.com/gradle/gradle/issues/4605 means that we can't rely on these lines being shown by Gradle.

[fmeum]

@florianGla @MarkusZoppelt FYI, this might hit you with Gradle.

[CodeLowSky]

Thanks for the info, this is working now with JAZZER_FUZZ=1, but how does your cmdline for testing look like? When I run the tests with gradle test it is putting the crash reports in the root dir and if I rerun without JAZZER_FUZZ set it ignores the inputs even if I move the crash files into the correct folder.

[fmeum]

You have to create a resource directory corresponding to your fuzz test class with Inputs appended for Jazzer to collect crashes in it and replay them when ordinary tests are running. In your case, creating src/test/resources/dummy/DummyFuzzTestInputs should work (I tested it locally).

[CodeLowSky]

This is my input dir

$ ls app/src/test/resources/dummy/DummyFuzzTestInputs/
crash  crash-48d5cf776df1ec92df4c006da22b18d3b6bba8ad  crash-d3adc5273397f1d045dff06fee4da09a60565212  crash-f90848556a0b80428788dd0cb58d0de8d9bde29d

And this is the test output with gradle clean test --info

DummyFuzzTest > dummyFuzz(FuzzedDataProvider) > dummy.DummyFuzzTest.dummyFuzz(FuzzedDataProvider)[1] STANDARD_OUT
    INFO: Loaded 163 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
    INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
    INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
    INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
    INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
    INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
    INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
    INFO: Loaded 50 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
    INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
    INFO: Loaded 76 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
    INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
    INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
    INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
    INFO: Instrumented dummy.DummyFuzzTest with custom hooks only (took 31 ms, size +0%)
    INFO: Instrumented dummy.App with custom hooks only (took 5 ms, size +0%)

Gradle Test Executor 5 finished executing tests.

> Task :app:test
Finished generating test XML results (0.003 secs) into: .\jazzer_test\dummy\app\build\test-results\test
Generating HTML test report...
Finished generating test html results (0.007 secs) into: .\jazzer_test\dummy\app\build\reports\tests\test
:app:test (Thread[Execution worker Thread 5,5,main]) completed. Took 3.921 secs.

BUILD SUCCESSFUL in 5s

How do you execute the tests?

[fmeum]

In my tests, the src directory is at the root of the project, not nested in an app module directory. Jazzer doesn't support this yet, but it probably should.

[fmeum]

I created #523 to track this. As a workaround, could you try to set the JUnit working directory to be the app subdirectory of your project root? That is the default behavior with Maven, but apparently not with Gradle.

[CodeLowSky]

Even if I move src to the project root or set app as project root, it's the same behaviour. But for now I can work with that, maybe I'll find a solution. Thanks a lot for your help!

[ghost]

@CodeLowSky - Thanks for working through this issue. A lot has changed with Jazzer and our technology since October 2022. I'm curious if you're still using it or wanted to explore alternative options? Ping me? david[dot]merian [at] code-intelligence[dot]com