search

JuliaLang / Microbenchmarks

Microbenchmarks comparing the Julia Programming language with other languages
https://julialang.org/benchmarks/
Other
88 stars 48 forks source link

java mandelbrot benchmark was broken #49

Closed michaellilltokiwa closed 2 years ago

mkitti commented 2 years ago

On master, I get the following results:

~/src/Microbenchmarks/java$ ./setup.sh
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< julialang.org:javaBenchmarks >--------------------
[INFO] Building javaBenchmarks 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ javaBenchmarks ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/mkitti/src/Microbenchmarks/java/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ javaBenchmarks ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ javaBenchmarks >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ javaBenchmarks <<<
[INFO]
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ javaBenchmarks ---
java,recursion_fibonacci,0.111934
java,parse_integers,0.543593
java,userfunc_mandelbrot,0.130648
java,recursion_quicksort,0.965371
java,iteration_pi_sum,10.644722
java,matrix_statistics,28.197627
java,matrix_multiply,918.574614
java,iteration_sinc_sum,0.122608
java,print_to_file,109.915085
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.274 s
[INFO] Finished at: 2022-02-21T22:52:00-05:00
[INFO] ------------------------------------------------------------------------

If I add the environment variable, MAVEN_OPTS="-ea" I get the following error.


~/src/Microbenchmarks/java$ export MAVEN_OPTS="-ea"
~/src/Microbenchmarks/java$ ./setup.sh
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< julialang.org:javaBenchmarks >--------------------
[INFO] Building javaBenchmarks 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ javaBenchmarks ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/mkitti/src/Microbenchmarks/java/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ javaBenchmarks ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ javaBenchmarks >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ javaBenchmarks <<<
[INFO]
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ javaBenchmarks ---
java,recursion_fibonacci,0.108510
java,parse_integers,0.569948
[WARNING]
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:745)
Caused by: java.lang.AssertionError: value was 14719
    at PerfPure.runBenchmarks (PerfPure.java:73)
    at PerfBLAS.main (PerfBLAS.java:16)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:745)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.775 s
[INFO] Finished at: 2022-02-21T22:52:12-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (default-cli) on project javaBenchmarks: An exception occured while executing the Java class. null: InvocationTargetException: value was 14719 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
mkitti commented 2 years ago

If I change the assertion to check for 14791, I now get the following:

~/src/Microbenchmarks/java$ ./setup.sh
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< julialang.org:javaBenchmarks >--------------------
[INFO] Building javaBenchmarks 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ javaBenchmarks ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/mkitti/src/Microbenchmarks/java/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ javaBenchmarks ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ javaBenchmarks >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ javaBenchmarks <<<
[INFO]
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ javaBenchmarks ---
java,recursion_fibonacci,0.093926
java,parse_integers,0.473236
java,userfunc_mandelbrot,0.112116
java,recursion_quicksort,0.851171
java,iteration_pi_sum,10.669577
java,matrix_statistics,28.768373
java,matrix_multiply,926.444263
java,iteration_sinc_sum,0.121479
java,print_to_file,105.345361
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.269 s
[INFO] Finished at: 2022-02-21T22:59:43-05:00
[INFO] ------------------------------------------------------------------------
mkitti commented 2 years ago

Pulling this branch:

java,recursion_fibonacci,0.110110
java,parse_integers,0.521227
java,userfunc_mandelbrot,0.176235
java,recursion_quicksort,0.899142
java,iteration_pi_sum,10.440428
java,matrix_statistics,27.929927
java,matrix_multiply,922.701459
java,iteration_sinc_sum,0.123223
java,print_to_file,103.658632
java,recursion_fibonacci,0.093568
java,parse_integers,0.513418
java,userfunc_mandelbrot,0.155771
java,recursion_quicksort,0.888630
java,iteration_pi_sum,10.436011
java,matrix_statistics,27.481112
java,matrix_multiply,923.766102
java,iteration_sinc_sum,0.123808
java,print_to_file,105.001560
java,recursion_fibonacci,0.143873
java,parse_integers,0.555857
java,userfunc_mandelbrot,0.164974
java,recursion_quicksort,1.169194
java,iteration_pi_sum,10.663690
java,matrix_statistics,27.559458
java,matrix_multiply,913.833593
java,iteration_sinc_sum,0.122547
java,print_to_file,102.976528

All the above benchmarks are via

$ java -version
openjdk version "1.8.0_152-release"
OpenJDK Runtime Environment (build 1.8.0_152-release-1056-b12)
OpenJDK 64-Bit Server VM (build 25.152-b12, mixed mode)

There does appear to be a performance hit on mandelbrot.

mkitti commented 2 years ago

I got up to Java 11 before maven broke. I will boot up a virtual machine later and retry.

My current recommendation is to split this into two pull requests. First submit the minimal changes needed to fix the assertion at 14791. Then submit the other changes.

michaellilltokiwa commented 2 years ago

I got up to Java 11 before maven broke. I will boot up a virtual machine later and retry.

My current recommendation is to split this into two pull requests. First submit the minimal changes needed to fix the assertion at 14791. Then submit the other changes.

Sorry, will not do that. But you can decline the PR and use the parts if you want.