JDK7 => JDK8 Performance Degradation

[mike-bresnahan]

I am using JMockit 1.7 and 1.37.

I recently upgraded our application from JDK7 to JDK8 and the JMockit-based testsuite is running about 2 times slower. The performance seems to progressively degrade as the testsuite runs. We have similar applications using different mocking frameworks that do not exhibit the same behavior.

I am aware of this discussion:

JMockit is roughly 2-3 times slower on Java8 than Java7 https://groups.google.com/forum/#!searchin/jmockit-users/JDK8%7Csort:date/jmockit-users/MCm97zsjjd4/WQPU_6PLCgAJ

I have been able to reproduce the issue discussed using the test app provided using JMockit 1.15, but not with 1.7 or 1.37. Also, the proposed work-around (-Xint) does not work for my issue. Because of these facts, I suspect my issue is a different issue than what is discussed in the discussion linked above. I am curious though why the issue discussed is not reproducible with JMockit 1.7 and 1.37, because it was described as a 64bit JVM issue not a JMockit issue. Background around that issue might give me some clues.

I have profiled the code using VisualVM but have not been able to find a smoking gun. The profile data suggests a general slowdown of every function profiled. While the garbage collection behavior is different with JDK8 vs JDK7, it is not obvious that it is slowing things down.

I have reproduced the issue on 2 different macbooks and also on a windows 7 laptop.

I apologize for a post here because I have not been able to positively identify the root cause being JMockit, however the JMockit Users google group appears to not allow new posts and I'm not aware of an alternative. I am still just looking for clues for what might be the root cause.

[rliesenfeld]

Ok. Let us know if you find more information about the cause.

[rliesenfeld]

I just performed some test runs of JMockit's main test suite (about 1100 tests) using Maven (surefire), on a Windows 7 32-bit machine. Ran on JDK 1.7.0_80 and on JDK 1.8.0_144, with everything else the same. The total time reported by Maven is a bit over 17 seconds, with both JDKs. So, there is no difference that I can see.

It's possible that in some very particular mocking scenario, there would be a non-negligible difference between JDKs 7 and 8. Maybe something like #390. In such a case, I believe the solution would to be review the tests in order to reduce the amount of mocking/faking (which is always a good idea anyway).

[mike-bresnahan]

I have managed to get it narrowed down to a simple test case. The root cause appears to be using a MockUp in the following way. On JDK7, this runs 100 tests in about 0.17s on my macbook. On JDK8 it takes about 0.61s.

MyClassTest.java:

package app;

import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;

public class MyClassTest {
    StuffService stuffService;
    MyClass myClass;

    @Before
    public void before() {
        myClass = new MyClass();
        new MockStuffService();
        stuffService = new StuffService();
        myClass.stuffService = stuffService;
    }

    @Test
    public void test() {
        Stuff stuff = myClass.getStuff();
        Assert.assertNotNull(stuff);
    }
}

MockStuffService.java:

package app;

import mockit.Mock;
import mockit.MockUp;

public class MockStuffService extends MockUp<StuffService> {
    @Mock
    public Stuff getStuff() {
        return new Stuff();
    }
}

PerfTest.java:

package app;

import org.junit.internal.TextListener;
import org.junit.runner.JUnitCore;

public class PerfTest {

    static public void main(String args[]) {
        System.out.println("java.version=" + System.getProperty("java.version"));
        int n = 100;
        Class<?>[] classes = new Class[n];
        for (int i = 0; i < n; ++i) {
            classes[i] = MyClassTest.class;
        }
        JUnitCore junit = new JUnitCore();
        junit.addListener(new TextListener(System.out));
        junit.run(classes);
    }
}

test3.tar.gz

[mike-bresnahan]

Interestingly, my test case runs 100 tests in 0.19s when -Xint is used. So this seems like the JDK8 x64 bug discussed on google groups. I'm very confused why -Xint has no apparent effect on my real test suite nor on the JMockitTest test case posted to google groups (unless used with jmockit 1.15).

[mike-bresnahan]

Actually I think I might understand after reading about the -Xint option. My real test suite probably doesn't benefit from -Xint because, while it might speed up parts of jmockit, it slows everything else down so much that there is little or no net result. The JMockitTest results could be explained by a difference in 1.7/1.37 vs 1.15 that mitigates the effect of the JDK8 bug.

[mike-bresnahan]

I ran my test case on JDK9 and it is about twice as slow as JDK8. It doesn't look like Oracle fixed the issue yet.

JDK7: 0.2s JDK8: 0.6s JDK9: 1.2s

[rliesenfeld]

Yep, it's slower on JDK 9 (see it on my 32-bit Windows 7 as well), which was not surprising. This isn't going to be "fixed", it's just how it is. And, of course, there is nothing to do about it on JMockit's side, since the timing differences occur by simply varying the exact JVM being used (JDK 7/8/9 x 32/64-bit x with or without "-Xint").