Regression tests cannot run some benchmarks in DaCapo 2006 on OpenJDK

[wks]

TL;DR: We should

• [ ] Switch to a newer DaCapo benchmark for OpenJDK. We will keep DaCapo 2006 for JikesRVM because it doesn't support newer versions.
• [ ] Mark failing tests red on the dashboard

As seen in https://www.mmtk.io/ci-perf-result/, some benchmarks have always been failing. Those include bloat, jython and sunflow.

bloat and jython can run normally on OpenJDK 8

wks@luna ~/o/dacapo> /usr/lib/jvm/java-8-openjdk/bin/java -jar dacapo-2006-10-MR2.jar bloat
===== DaCapo bloat starting =====
Optimized with: EDU.purdue.cs.bloat.optimize.Main -only EDU.purdue.cs.bloat.trans -pre -dce -diva -prop -stack-alloc -peel-loops all -f EDU.purdue.cs.bloat.trans.CompactArrayInitializer ./scratch/optimizedcode
===== DaCapo bloat PASSED in 782 msec =====
wks@luna ~/o/dacapo> /usr/lib/jvm/java-8-openjdk/bin/java -jar dacapo-2006-10-MR2.jar jython
*sys-package-mgr*: processing new jar, '/home/wks/opt/dacapo/dacapo-2006-10-MR2.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/resources.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/rt.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/jsse.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/jce.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/charsets.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/jfr.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/dnsns.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunjce_provider.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/localedata.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunpkcs11.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/cldrdata.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/sunec.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/nashorn.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/zipfs.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-8-openjdk/jre/lib/ext/jaccess.jar'
===== DaCapo jython starting =====
-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using Python 2.2a1
* Python version doesn't support garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

Calibrating tests. Please wait...

Running 1 round(s) of the suite at warp factor 20:

* Round 1 done in 0.976 seconds.

-------------------------------------------------------------------------------
Benchmark: 2024-10-30 14:48:52
-------------------------------------------------------------------------------

    Rounds: 1
    Warp:   20
    Timer:  time.time

Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     10ms     10ms    0.04us    0.000ms
           BuiltinMethodLookup:     24ms     24ms    0.05us    0.000ms
                 CompareFloats:     12ms     12ms    0.02us    0.000ms
         CompareFloatsIntegers:     12ms     12ms    0.03us    0.000ms
               CompareIntegers:     18ms     18ms    0.02us    0.000ms
        CompareInternedStrings:     20ms     20ms    0.03us    0.000ms
                  CompareLongs:     13ms     13ms    0.02us    0.000ms
                CompareStrings:     14ms     14ms    0.03us    0.000ms
                CompareUnicode:     12ms     12ms    0.03us    0.000ms
                 ConcatStrings:     45ms     45ms    0.18us    0.000ms
                 ConcatUnicode:     24ms     24ms    0.16us    0.000ms
               CreateInstances:     14ms     14ms    0.25us    0.000ms
            CreateNewInstances:     29ms     29ms    0.69us    0.000ms
       CreateStringsWithConcat:     35ms     35ms    0.07us    0.000ms
       CreateUnicodeWithConcat:      8ms      8ms    0.04us    0.000ms
                  DictCreation:     16ms     16ms    0.08us    0.000ms
             DictWithFloatKeys:     16ms     16ms    0.04us    0.000ms
           DictWithIntegerKeys:     21ms     21ms    0.03us    0.000ms
            DictWithStringKeys:     16ms     16ms    0.03us    0.000ms
                      ForLoops:     26ms     26ms    2.08us    0.000ms
                    IfThenElse:     21ms     21ms    0.03us    0.000ms
                   ListSlicing:      5ms      5ms    0.71us    0.000ms
                NestedForLoops:     22ms     22ms    0.03us    0.000ms
          NormalClassAttribute:     17ms     17ms    0.03us    0.000ms
       NormalInstanceAttribute:     15ms     15ms    0.02us    0.000ms
           PythonFunctionCalls:     15ms     15ms    0.09us    0.000ms
             PythonMethodCalls:     15ms     15ms    0.13us    0.000ms
                     Recursion:     13ms     13ms    0.52us    0.000ms
                  SecondImport:     16ms     16ms    0.32us    0.000ms
           SecondPackageImport:     21ms     21ms    0.42us    0.000ms
         SecondSubmoduleImport:     17ms     17ms    0.34us    0.000ms
       SimpleComplexArithmetic:     16ms     16ms    0.04us    0.000ms
        SimpleDictManipulation:     17ms     17ms    0.03us    0.000ms
         SimpleFloatArithmetic:     21ms     21ms    0.03us    0.000ms
      SimpleIntFloatArithmetic:     20ms     20ms    0.03us    0.000ms
       SimpleIntegerArithmetic:     19ms     19ms    0.03us    0.000ms
        SimpleListManipulation:     17ms     17ms    0.03us    0.000ms
          SimpleLongArithmetic:     19ms     19ms    0.06us    0.000ms
                    SmallLists:     14ms     14ms    0.04us    0.000ms
                   SmallTuples:     28ms     28ms    0.10us    0.000ms
         SpecialClassAttribute:     16ms     16ms    0.03us    0.000ms
      SpecialInstanceAttribute:     15ms     15ms    0.03us    0.000ms
                StringMappings:     32ms     32ms    0.25us    0.000ms
              StringPredicates:     22ms     22ms    0.06us    0.000ms
                 StringSlicing:     14ms     14ms    0.05us    0.000ms
                     TryExcept:      6ms      6ms    0.01us    0.000ms
                TryRaiseExcept:     52ms     52ms    1.63us    0.000ms
                  TupleSlicing:     10ms     10ms    0.08us    0.000ms
               UnicodeMappings:     40ms     40ms    2.22us    0.000ms
             UnicodePredicates:     20ms     20ms    0.07us    0.000ms
                UnicodeSlicing:     16ms     16ms    0.07us    0.000ms
-------------------------------------------------------------------------------
Totals:                            976ms    976ms

===== DaCapo jython PASSED in 1137 msec =====

But they will fail on OpenJDK 11

wks@luna ~/o/dacapo> /usr/lib/jvm/java-11-openjdk/bin/java -jar dacapo-2006-10-MR2.jar bloat
===== DaCapo bloat starting =====
java.lang.RuntimeException: Class not found: java/lang/Object
java.lang.RuntimeException: Class not found: java/lang/Object
        at EDU.purdue.cs.bloat.editor.ClassHierarchy.addClass(ClassHierarchy.java:386)
        at EDU.purdue.cs.bloat.editor.ClassHierarchy.addClassNamed(ClassHierarchy.java:116)
        at EDU.purdue.cs.bloat.editor.ClassHierarchy.getExtendsNode(ClassHierarchy.java:308)
        at EDU.purdue.cs.bloat.editor.ClassHierarchy.addClass(ClassHierarchy.java:401)
        at EDU.purdue.cs.bloat.editor.ClassHierarchy.addClassNamed(ClassHierarchy.java:116)
        at EDU.purdue.cs.bloat.context.PersistentBloatContext.addClasses(PersistentBloatContext.java:116)
        at EDU.purdue.cs.bloat.context.CachingBloatContext.<init>(CachingBloatContext.java:70)
        at EDU.purdue.cs.bloat.optimize.Main.main(Main.java:413)
        at dacapo.bloat.BloatHarness.iterate(BloatHarness.java:25)
        at dacapo.Benchmark.run(Benchmark.java:126)
        at dacapo.TestHarness.runBenchmark(TestHarness.java:302)
        at dacapo.TestHarness.main(TestHarness.java:242)
        at Harness.main(Harness.java:5)
wks@luna ~/o/dacapo [255]> /usr/lib/jvm/java-11-openjdk/bin/java -jar dacapo-2006-10-MR2.jar jython
*sys-package-mgr*: processing new jar, '/home/wks/opt/dacapo/dacapo-2006-10-MR2.jar'
===== DaCapo jython starting =====
Traceback (innermost last):
  File "./scratch/jython/pybench/pybench.py", line 38, in ?
  File "/home/wks/opt/dacapo/./scratch/jython/pybench/CommandLine.py", line 130, in ?
AttributeError: class 'org.python.modules.os' has no attribute 'path'

They are likely due to the API change between Java 8 and Java 11.

sunflow has been failing on all versions of OpenJDK with the same error:

wks@luna ~/o/dacapo [255]> /usr/lib/jvm/java-8-openjdk/bin/java -jar dacapo-2006-10-MR2.jar sunflow
java.lang.ClassNotFoundException: dacapo.sunflow.SunflowHarness
java.lang.ClassNotFoundException: dacapo.sunflow.SunflowHarness
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at dacapo.TestHarness.findClass(TestHarness.java:414)
        at dacapo.TestHarness.main(TestHarness.java:241)
        at Harness.main(Harness.java:5)

and a slightly different message in JikesRVM:

wks@luna ~/o/dacapo [255]> setarch -R ~/projects/mmtk-github/jikesrvm/dist/RBaseBaseSemiSpace_x86_64_m32-linux/rvm -Xm{s,x}75M -jar dacapo-2006-10-MR2.jar sunflow
[2024-10-30T06:52:46Z INFO  mmtk::memory_manager] Initialized MMTk with SemiSpace (FixedHeapSize(78643200))
java.lang.ClassNotFoundException: dacapo.sunflow.SunflowHarness not found in SystemAppCL
java.lang.ClassNotFoundException: dacapo.sunflow.SunflowHarness not found in SystemAppCL
   at java.net.URLClassLoader.findClass(URLClassLoader.java:531)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:341)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:293)
   at org.jikesrvm.classloader.TypeReference.resolveInternal(TypeReference.java:815)
   at org.jikesrvm.classloader.TypeReference.resolve(TypeReference.java:803)
   at java.lang.Class.forNameInternal(Class.java:809)
   at java.lang.Class.forName(Class.java:768)
   at dacapo.TestHarness.findClass(TestHarness.java:414)
   at dacapo.TestHarness.main(TestHarness.java:241)
   at Harness.main(Harness.java:5)

Since DaCapo 2006 was released decades ago, it is unsurprising that some benchmarks cannot run on modern JVMs.

For OpenJDK, it's better to switch to a new DaCapo version, such as Chopin, because it uses modern workloads that are more suitable for modern JVMs.

We should also mark failing tests red on the dashboard so that it will clear indicate that there is a failure.