search

CodeIntelligenceTesting / jazzer

Coverage-guided, in-process fuzzing for the JVM
https://code-intelligence.com
Other
1k stars 133 forks source link

junit tests don't generate java code to reproduce exceptions #607

Open freedom1b2830 opened 1 year ago

freedom1b2830 commented 1 year ago

uname -a Linux archlinux 6.1.5-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 12 Jan 2023 22:42:33 +0000 x86_64 GNU/Linux

mvn -version

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/maven
Java version: 19.0.1, vendor: N/A, runtime: /usr/lib/jvm/java-19-openjdk
Default locale: ru_RU, platform encoding: UTF-8
OS name: "linux", version: "6.1.5-arch2-1", arch: "amd64", family: "unix"

mvn test

[INFO] Scanning for projects...
[INFO] 
[INFO] -------------------------< freedom1b2830:test >-------------------------
[INFO] Building test 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ test ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ test ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/user_dev_new/eclipse-workspace/test/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ test ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ test ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/user_dev_new/eclipse-workspace/test/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M7:test (default-test) @ test ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.AutofuzzFuzzTest
INFO: Loaded 159 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
INFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
INFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
INFO: Instrumented com.example.AutofuzzFuzzTest (took 186 ms, size +11%)
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3546565980
INFO: Loaded 1 modules   (512 inline 8-bit counters): 512 [0x7fab04931f30, 0x7fab04932130), 
INFO: Loaded 1 PC tables (512 PCs): 512 [0x7fab048b0b70,0x7fab048b2b70), 
INFO:        5 files found in /home/user_dev_new/eclipse-workspace/test/.cifuzz-corpus/com.example.AutofuzzFuzzTest
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 5 min: 1b max: 10b total: 27b rss: 674Mb
INFO: Instrumented com.example.ExploreMe (took 38 ms, size +111%)
#6      INITED cov: 15 ft: 15 corp: 5/27b exec/s: 0 rss: 674Mb
#32768  pulse  cov: 15 ft: 15 corp: 5/27b lim: 331 exec/s: 16384 rss: 674Mb
#65536  pulse  cov: 15 ft: 15 corp: 5/27b lim: 659 exec/s: 21845 rss: 674Mb
MS: 2 CMP-EraseBytes- DE: "jaz.Zer"-; base unit: 872ba8e6cdb8fd1598d67a39d2f93875fa39407a
0x40,0x6a,0x61,0x7a,0x2e,0x5a,0x65,0x72,0x69,0x73,0xa,0x58,0x58,0x58,0xce,0x58,
@jaz.Zeris\012XXX\316X
artifact_prefix='/home/user_dev_new/eclipse-workspace/test/'; Test unit written to /home/user_dev_new/eclipse-workspace/test/crash-b6d7d4e892cc5af63b34f111bc62c8a4a26325b8
Base64: QGphei5aZXJpcwpYWFjOWA==
Done 118343 runs in 4 second(s)
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 23.201 s <<< FAILURE! - in com.example.AutofuzzFuzzTest
[ERROR] com.example.AutofuzzFuzzTest.myFuzzTest(FuzzedDataProvider)[1]  Time elapsed: 5.461 s  <<< ERROR!
com.code_intelligence.jazzer.api.FuzzerSecurityIssueHigh: 
Remote Code Execution
Unrestricted class loading based on externally controlled data may allow
remote code execution depending on available classes on the classpath.
        at jaz.Zer.<clinit>(Zer.java:54)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:390)
        at java.base/java.lang.Class.forName(Class.java:381)
        at com.example.ExploreMe.exploreMe(ExploreMe.java:13)
        at com.example.AutofuzzFuzzTest.myFuzzTest(AutofuzzFuzzTest.java:21)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   AutofuzzFuzzTest.myFuzzTest:21 » FuzzerSecurityIssueHigh Remote Code Execution
Unrestricted class loading based on externally controlled data may allow
remote code execution depending on available classes on the classpath.
[INFO] 
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 1
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.589 s
[INFO] Finished at: 2023-01-20T09:39:33Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M7:test (default-test) on project test: 
[ERROR] 
[ERROR] Please refer to /home/user_dev_new/eclipse-workspace/test/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

ls -la

drwxr-xr-x. 1 user_dev_new user_dev_new  234 янв 20 09:39 .
drwxr-xr-x. 1 user_dev_new user_dev_new  526 янв  5 16:34 ..
drwxr-xr-x. 1 user_dev_new user_dev_new   56 дек 31 11:33 .cifuzz-corpus
-rw-r--r--. 1 user_dev_new user_dev_new 1478 дек 28 10:09 .classpath
-rw-r--r--. 1 user_dev_new user_dev_new   16 янв 20 09:39 crash-b6d7d4e892cc5af63b34f111bc62c8a4a26325b8
drwxr-xr-x. 1 user_dev_new user_dev_new  144 янв  2 12:10 .git
-rw-r--r--. 1 user_dev_new user_dev_new   39 дек 31 10:46 .gitignore
-rw-r--r--. 1 user_dev_new user_dev_new 1443 янв 20 09:38 pom.xml
-rw-r--r--. 1 user_dev_new user_dev_new  533 дек 28 08:07 .project
drwxr-xr-x. 1 user_dev_new user_dev_new  168 дек 28 10:09 .settings
drwxr-xr-x. 1 user_dev_new user_dev_new   16 дек 28 08:07 src
drwxr-xr-x. 1 user_dev_new user_dev_new  172 янв 20 09:39 target

find src/ -type f

src/main/java/com/example/ExploreMe.java
src/test/java/com/example/AutofuzzFuzzTest.java

pom.xml

        <properties>
                <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
                <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
                <maven.compiler.source>11</maven.compiler.source>
                <maven.compiler.target>11</maven.compiler.target>
        </properties>
        <dependencies>
                <dependency>
                        <groupId>com.google.code.findbugs</groupId>
                        <artifactId>jsr305</artifactId>
                        <version>3.0.2</version>
                </dependency>
                <dependency>
                        <groupId>org.junit.jupiter</groupId>
                        <artifactId>junit-jupiter</artifactId>
                        <version>5.9.2</version>
                </dependency>
                <dependency>
                        <groupId>com.code-intelligence</groupId>
                        <artifactId>jazzer-junit</artifactId>
                        <version>0.14.0</version>
                </dependency>
        </dependencies>

        <build>
                <plugins> 
                        <plugin>
                                <artifactId>maven-surefire-plugin</artifactId>
                                <version>3.0.0-M7</version>
                        </plugin>
                </plugins>
                <testResources>
                        <testResource>
                                <directory>${project.basedir}/src/test/resources</directory>
                        </testResource>
                </testResources>
        </build>
bertschneider commented 1 year ago

Correct, this is currently not supported by the JUnit integration.

Findings are stored in the corpus directory and used in subsequent regression mode runs, which is a nicer way to reproduce/debug findings to begin with. Have you tried that out?

freedom1b2830 commented 1 year ago

No, I didn't use it. Can you show how to use these files ?

bertschneider commented 1 year ago

How to use the JUnit integration is described at Using Jazzer via JUnit 5, but you saw that as you're using the integration already.

Point 5. describes how to use found issues in the regression test mode. In this mode you can start the test like a normal JUnit test from within your IDE, set breakpoints and basically do everything you could do in a normal unit test. You could also only execute the test of a specific input to easily reproduce the found issue. How to do that is IDE dependent, though.

freedom1b2830 commented 1 year ago

Eclipse IDE for Java Developers (includes Incubating components) Version: 2022-12 (4.26.0) Build id: 20221201-1913

without JAZZER_FUZZ=1 1)mvn clean install 2)eclipse run test

java.lang.IllegalStateException: Failed to run Agent.install
    at com.code_intelligence.jazzer.agent.AgentInstaller.install(AgentInstaller.java:49)
    at com.code_intelligence.jazzer.junit.FuzzTestArgumentsProvider.configureAndInstallAgent(FuzzTestArgumentsProvider.java:71)
    at com.code_intelligence.jazzer.junit.FuzzTestArgumentsProvider.provideArguments(FuzzTestArgumentsProvider.java:80)
    at org.junit.jupiter.params.ParameterizedTestExtension.arguments(ParameterizedTestExtension.java:145)
    at org.junit.jupiter.params.ParameterizedTestExtension.lambda$provideTestTemplateInvocationContexts$2(ParameterizedTestExtension.java:90)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:110)
    at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:44)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:95)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:91)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:60)
    at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
    Suppressed: org.junit.platform.commons.PreconditionViolationException: Configuration error: You must configure at least one set of arguments for this @ParameterizedTest
        at org.junit.platform.commons.util.Preconditions.condition(Preconditions.java:299)
        at org.junit.jupiter.params.ParameterizedTestExtension.lambda$provideTestTemplateInvocationContexts$5(ParameterizedTestExtension.java:98)
        at java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:279)
        ... 56 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.code_intelligence.jazzer.agent.AgentInstaller.install(AgentInstaller.java:46)
    ... 72 more
Caused by: java.lang.VerifyError
    at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
    at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
    at com.code_intelligence.jazzer.agent.Agent.installInternal(Agent.kt:148)
    at com.code_intelligence.jazzer.agent.Agent.installInternal$default(Agent.kt:36)
    at com.code_intelligence.jazzer.agent.Agent.install(Agent.kt:33)
    ... 77 more
fmeum commented 1 year ago

This should be worked around quite effectively by https://github.com/CodeIntelligenceTesting/jazzer/commit/48ff37c56954ca50c4439f7023006026c22c4057, even though I can't tell what the root cause is.

@freedom1b2830 Please test with the next Jazzer release (not out yet).