search

rohanpadhye / JQF

JQF + Zest: Coverage-guided semantic fuzzing for Java.
BSD 2-Clause "Simplified" License
666 stars 112 forks source link

having problem fuzzing - due to clojure loading #90

Open guyarb opened 4 years ago

guyarb commented 4 years ago

Hi,

I'm implementing a small java project which load and run clojure scripts on runtime. I works fine.

I want to use jqf:fuzz to fuzz my application. So i copied the files from the compiler tutorial of jqf and google closure compiler and modified it to my specs.

The project exists in here.

To run my program:

mvn package
java -jar target/zest-tutorial-1.0-SNAPSHOT.jar com.comp.src.app.app -main

to fuzz:

mvn jqf:fuzz -Dclass=examples.CompilerTest -Dmethod=testWithString

the output:

Semantic Fuzzing with Zest                                                                                                                                                      
--------------------------                                                                                                                                                      

Test name:            examples.CompilerTest#testWithString                                                                                                                      
Results directory:    /home/<username>/java-clojure-example/target/fuzz-results/examples.CompilerTest/testWithString                                                              
Elapsed time:         2s (no time limit)                                                                                                                                        
Number of executions: 5,158                                                                                                                                                     
Valid inputs:         0 (0.00%)                                                                                                                                                 
Cycles completed:     0                                                                                                                                                         
Unique failures:      3                                                                                                                                                         
Queue size:           0 (0 favored last cycle)                                                                                                                                  
Current parent input: <seed>                                                                                                                                                    
Execution speed:      6,613/sec now | 1,909/sec overall                                                                                                                         
Total coverage:       0 branches (0.00% of map)                                                                                                                                 
Valid coverage:       0 branches (0.00% of map)

the reproduction:

mvn jqf:repro -Dclass=examples.CompilerTest -Dmethod=testWithString -Dinput=target/fuzz-results/examples.CompilerTest/testWithString/failures/id_000000

[INFO] Scanning for projects...                                                                                                                                                 
[INFO]                                                                                                                                                                          
[INFO] -----------------------< examples:zest-tutorial >-----------------------                                                                                                 
[INFO] Building zest-tutorial 1.0-SNAPSHOT                                                                                                                                      
[INFO] --------------------------------[ jar ]---------------------------------                                                                                                 
[INFO]                                                                                                                                                                          
[INFO] --- jqf-maven-plugin:1.4:repro (default-cli) @ zest-tutorial ---                                                                                                         
.id_000000 ::= FAILURE (java.lang.ExceptionInInitializerError)                                                                                                                  
E                                                                                                                                                                               
Time: 1.039                                                                                                                                                                     
There was 1 failure:                                                                                                                                                            
1) testWithString(examples.CompilerTest)                                                                                                                                        
java.lang.ExceptionInInitializerError                                                                                                                                           
        at clojure.java.api.Clojure.<clinit>(Clojure.java)                                                                                                                      
        at examples.CompilerTest.testWithString(CompilerTest.java)                                                                                                              
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                          
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                        
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                     
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)                                                                                 
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)                                                                                  
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)                                                                                   
        at edu.berkeley.cs.jqf.fuzz.junit.TrialRunner$1.evaluate(TrialRunner.java:59)                                                                                           
        at edu.berkeley.cs.jqf.fuzz.junit.TrialRunner.run(TrialRunner.java:65)                                                                                                  
        at edu.berkeley.cs.jqf.fuzz.junit.quickcheck.FuzzStatement.evaluate(FuzzStatement.java:168)                                                                             
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)                                                                                                        
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)                                                                                    
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)                                                                                    
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)                                                                                                          
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)                                                                                                      
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)                                                                                                    
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)                                                                                                      
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)                                                                                                     
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)                                                                                                            
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)                                                                                                                   
        at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:180)                                                                                             
        at edu.berkeley.cs.jqf.fuzz.junit.GuidedFuzzing.run(GuidedFuzzing.java:126)                                                                                             
        at edu.berkeley.cs.jqf.plugin.ReproGoal.execute(ReproGoal.java:205) 
        at edu.berkeley.cs.jqf.plugin.ReproGoal.execute(ReproGoal.java:205)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class, clojure/core.clj or clojure/core.cljc on classpath.
        at clojure.lang.RT.load(RT.java)
        at clojure.lang.RT.load(RT.java)
        at clojure.lang.RT.<clinit>(RT.java)
        ... 47 more

FAILURES!!!
Tests run: 1,  Failures: 1

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.781 s
[INFO] Finished at: 2020-04-13T15:51:37+03:00

i'm running on ubuntu 18.04 64 bit, java 8

Thanks!

guyarb commented 4 years ago

I'm able to run jqf-zest manually by:

<JQF-PATH>/bin/jqf-zest -c .:$(<JQF-PATH>/scripts/classpath.sh):target/test-classes:$(cd src/main/clojure/com/comp && lein with-profile default classpath) examples.CompilerTest testWithString

and getting the following output:

Semantic Fuzzing with Zest
--------------------------

Test name:            examples.CompilerTest#testWithString
Results directory:    /home/<username>/java-clojure-example/fuzz-results
Elapsed time:         6s (no time limit)
Number of executions: 982
Valid inputs:         982 (100.00%)
Cycles completed:     0
Unique failures:      0
Queue size:           2 (0 favored last cycle)
Current parent input: 1 (favored) {1/720 mutations}
Execution speed:      756/sec now | 141/sec overall
Total coverage:       745 branches (1.14% of map)
Valid coverage:       745 branches (1.14% of map)

but again, cannot do it with mvn jqf:fuzz

rohanpadhye commented 4 years ago

Thanks for the report! I see the root cause with the Maven plugin is the following:

Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class, clojure/core.clj or clojure/core.cljc on classpath.
        at clojure.lang.RT.load(RT.java)
        at clojure.lang.RT.load(RT.java)
        at clojure.lang.RT.<clinit>(RT.java)

I don't know much about clojure to guess where these classes should be. How does the JVM/Maven usually find the Clojure runtime classes? For example, if you simply run mvn test instead of mvn jqf:fuzz, then does it run your @Fuzz methods without any issue? Note that simply running mvn test will cause all your fuzz targets to be run with a fixed number (I think 100) of randomly generated trials (without any coverage feedback). However, it is a good sanity check to make sure that your fuzz target and generators etc. are properly setup.

guyarb commented 4 years ago

Thanks for the reply!

The mvn test command ran successfully and managed to find the clojure jar. I'm suspecting the problem lays within the classpath generated for (or by) the jqf:fuzz. How can I override or add my own paths to the classpath of jqf:fuzz?

user@user:java-clojure-example ‹master*›$ mvn test
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< examples:zest-tutorial >-----------------------
[INFO] Building zest-tutorial 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ zest-tutorial ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/<username>/java-clojure-example/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ zest-tutorial ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/<username>/java-clojure-example/target/classes
[INFO] 
[INFO] --- clojure-maven-plugin:1.8.4:compile (compile 1) @ zest-tutorial ---
Compiling com.comp.src.app.app to /home/<username>/java-clojure-example/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ zest-tutorial ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/<username>/java-clojure-example/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ zest-tutorial ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/<username>/java-clojure-example/target/test-classes
[INFO] 
[INFO] --- clojure-maven-plugin:1.8.4:testCompile (compile 2) @ zest-tutorial ---
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ zest-tutorial ---
[INFO] Surefire report directory: /home/<username>/java-clojure-example/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running examples.CompilerTest
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.993 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.796 s
[INFO] Finished at: 2020-04-16T07:24:30+03:00
[INFO] ------------------------------------------------------------------------
rohanpadhye commented 4 years ago

That's very interesting.

The JQF Maven plugin just reuses whatever classpath mvn test would ordinarily get: https://github.com/rohanpadhye/JQF/blob/008432b79aeda0b68dd709931e19c988faf21ed0/maven-plugin/src/main/java/edu/berkeley/cs/jqf/plugin/FuzzGoal.java#L280

The TEST resolution scope includes "compile + system + provided + runtime + test dependencies".

How can I override or add my own paths to the classpath of jqf:fuzz?

Normally, the answer is to add appropriate classpath entries to your pom.xml under <dependencies>. However, there seems to be some disparity between the classpath that Maven gets when running mvn test and when running a plugin such as mvn jqf:fuzz. I am quite confused to be honest. Can you tell me how the clojure runtime is specified in your pom.xml for other non-JQF purposes?

guyarb commented 4 years ago

I've added the clojure library as a dependency in my pom.xml

<dependency>
  <groupId>org.clojure</groupId>
  <artifactId>clojure</artifactId>
  <version>1.10.1</version>
</dependency>

I've tried debugging the plugin and see the content of classpathElements and saw it has the jar of clojure /home/<username>/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar Furthermore, I used the maven-dependency-plugin to copy and unpack the clojure dependency to the target/test-classes directory, but still getting the failure as above.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <version>2.7</version>
  <executions>
    <execution>
      <id>default-cli</id>
      <phase>test-compile</phase>
      <goals>
        <goal>unpack-dependencies</goal>
      </goals>
      <configuration>
        <outputDirectory>target/test-classes</outputDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>

Another thing I tried to do is to print in the test the classpath using the following code:

private void printClasspath() {
    ClassLoader cl = ClassLoader.getSystemClassLoader();
    URL[] urls = ((URLClassLoader)cl).getURLs();

    for(URL url: urls){
        System.out.println(url.getFile());
    }
}

And I get /usr/share/java/plexus-classworlds.jar as the single result