PermGen out-of-memory, EasyMock and JUnit runner interactions

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run a large set of mocking unit tests that use @PrepareForTest.

What is the expected output? What do you see instead?

Result is a PermGen OutOfMemoryException.

What version of the product are you using? On what operating system?

PowerMock 1.4.10 which has many of the fixes for memory issues, especially 
Issue 207.

EasyMock 3.0, with a patched ClassProxyFactory as attached to EasyMock issue: 
Memory leaks in EasyMock - ID: 3145906 (referred to from Issue 207).

OS (Windows 32 bit/Linux 64 bit) makes some difference to how large your 
-XXMaxPermSize needs to be to avoid the OOM. However, with large enough set of 
tests (like we have) the number becomes infeasible (>1G on 32-bit for example).

Please provide any additional information below.

I used Eclipse MAT to analyse a number of heap dumps to investigate this issue. 
There are obvious complications - the tests are run as  "plugin tests", hence 
use OSGi classloading.

After the fixes and patches above, it came down to small set of class 
references that were preventing the garbage collection of MockClassLoader 
instances. These class references are the values in the @PrepareForTest, 
@RunWith annotations themselves! The culprit is the JUnit Description class 
with contains an array of annotations from the test class. With the PowerMock 
runner (we're using the Junit44 delegate variant), these annotations are 
created with the MockClassLoader, hence the class references are from *this* 
class loader.  The Description instances were then being collected by the 
Eclipse JDT JUnit runner to track the tests that had been run - presumably 
other runners similarly keep track of tests this way.

The workaround/fix we've implemented is to use Whitebox to set the fAnnotations 
field in the Description for the test after the run. We already had subclassed 
the PowerMockRunner, so it was easy to do this.

    @Override
    public void run(final RunNotifier notifier) {
        Description description = getDescription();
        try {
            super.run(notifier);
        }
        finally {
            Whitebox.setInternalState(description, "fAnnotations", new Annotation[] {});
        }
    }

Note that getDescription() will NPE if called after super.run(). PowerMock 
already was trying to free references.

A more permanent fix should be possible in PowerMock doing something similar.

Original issue reported on code.google.com by dmeibu...@gmail.com on 7 Sep 2011 at 12:56

[GoogleCodeExporter]

Hi, 

Thanks A LOT for you're detailed description and the time it must have taken 
you to analyze this. I'll add the "try-finally" fix to PowerMock soon.

Awesome work! Thanks!

Original comment by johan.ha...@gmail.com on 10 Sep 2011 at 6:43

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

I've now applied it to trunk. Please verify whether it works. Thanks again for 
your help!

Original comment by johan.ha...@gmail.com on 14 Sep 2011 at 8:10

• Changed state: Fixed

[GoogleCodeExporter]

Hi,

Thanks to the reporter for the good job. May I know in which release (when) 
this fix will be available for those not working with the trunk version?

Thanks you
Rodrigue

Original comment by rlag...@gmail.com on 14 Sep 2011 at 3:38

[GoogleCodeExporter]

It will be available in the next release for sure but it'll probably take a 
while before it's released :/

Original comment by johan.ha...@gmail.com on 14 Sep 2011 at 5:46

[GoogleCodeExporter]

Hi Johan,

I just tested the quick fix suggested by Sprynter, and it doesn't work. Did you 
use exact the same solution?.

Thank you

Original comment by rlag...@gmail.com on 15 Sep 2011 at 12:59

[GoogleCodeExporter]

Note that you also need the EasyMock patch mentioned above. It is crucial.

The fix here to PowerMock is highly dependent on the JUnit runner being used. 
It fixes a reference leak that becomes apparent with the Eclipse JDT runner 
(specifically, Eclipse 3.5.x - not sure about other versions or other runners). 
In our specific case, this runner was being used with the Maven Tycho plugin.

Original comment by dmeibu...@gmail.com on 15 Sep 2011 at 4:38

[GoogleCodeExporter]

Yeah I know. Unfortunately there's not much I can do about EasyMock. We should 
try to bring attention to the issue by upvoting the issue if possible and talk 
to the EasyMock guys to patch it.

Original comment by johan.ha...@gmail.com on 15 Sep 2011 at 6:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi Sprynter and Johan,

And where do I find the path?

Thank you

Original comment by rlag...@gmail.com on 15 Sep 2011 at 11:36

[GoogleCodeExporter]

EasyMock patch: 
http://sourceforge.net/tracker/?func=detail&aid=3145906&group_id=82958&atid=5678
37

Original comment by dmeibu...@gmail.com on 15 Sep 2011 at 11:53

[GoogleCodeExporter]

Hi Sprynter and Johan,

just want to thank you so much for the awesome work on this issue. Your 
solutions work. I just applied the patch manually on easymock-3.0 source code. 

You saved me a lot of time. I'm glad to close this problem (I was fighting on 
it since 5 days without success). :-)

Thank you.

Original comment by rlag...@gmail.com on 16 Sep 2011 at 12:41

[GoogleCodeExporter]

Glad it works for you. Perhaps you could make the EasyMock community aware of 
this by sending them an e-mail on their mailing-list.

Original comment by johan.ha...@gmail.com on 16 Sep 2011 at 12:51

[GoogleCodeExporter]

Hi,

I still have problems with the last version of powermock-mockito-1.4.11-full. 
In change logs says that this is issue is fixed ( Fixed an OutOfMemory issue in 
PowerMockRunner (issue 346) ) but I still have the same problem. This is the 
stack trace:

java.lang.OutOfMemoryError: PermGen space
    at org.mockito.internal.stubbing.defaultanswers.ReturnsMoreEmptyValues.<init>(ReturnsMoreEmptyValues.java:47)
    at org.mockito.internal.stubbing.defaultanswers.ReturnsSmartNulls.<init>(ReturnsSmartNulls.java:60)
    at org.mockito.Answers.<clinit>(Answers.java:28)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at java.lang.Class.getEnumConstantsShared(Class.java:2942)
    at java.lang.Class.enumConstantDirectory(Class.java:2963)
    at java.lang.Enum.valueOf(Enum.java:191)
    at sun.reflect.annotation.AnnotationParser.parseEnumValue(AnnotationParser.java:413)
    at sun.reflect.annotation.AnnotationParser.parseMemberValue(AnnotationParser.java:278)
    at java.lang.reflect.Method.getDefaultValue(Method.java:720)
    at sun.reflect.annotation.AnnotationType.<init>(AnnotationType.java:99)
    at sun.reflect.annotation.AnnotationType.getInstance(AnnotationType.java:66)
    at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:202)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
    at java.lang.reflect.Field.declaredAnnotations(Field.java:1014)
    at java.lang.reflect.Field.getAnnotation(Field.java:1000)
    at java.lang.reflect.AccessibleObject.isAnnotationPresent(AccessibleObject.java:168)
    at org.powermock.reflect.internal.matcherstrategies.FieldAnnotationMatcherStrategy.matches(FieldAnnotationMatcherStrategy.java:37)
    at org.powermock.reflect.internal.WhiteboxImpl.findAllFieldsUsingStrategy(WhiteboxImpl.java:535)
    at org.powermock.reflect.internal.WhiteboxImpl.getFieldsAnnotatedWith(WhiteboxImpl.java:2343)
    at org.powermock.reflect.internal.WhiteboxImpl.getFieldsAnnotatedWith(WhiteboxImpl.java:2326)
    at org.powermock.reflect.Whitebox.getFieldsAnnotatedWith(Whitebox.java:568)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:70)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:284)
    at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
    at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:209)
    at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:148)

Original comment by amanriqu...@gmail.com on 17 Jan 2012 at 1:32

[GoogleCodeExporter]

Some OOM issues have been resolved and EasyMock has accepted one of my patches 
that solved an OOM issue in EasyMock itself. There may be more. Please help out!

Original comment by johan.ha...@gmail.com on 17 Jan 2012 at 3:39

[GoogleCodeExporter]

Did you get the OOM issue in version 1.4.10 as well?

Original comment by johan.ha...@gmail.com on 17 Jan 2012 at 3:40

[GoogleCodeExporter]

Yes, the same issue in 1.4.10 than in 1.4.11. I run 458 unit tests that are 
mocking a lot of things ... each one of them use @PrepareForTest... If I run 
this tests in diferent executions there is no problem ... Should I apply some 
other patch?? or there is still a lot of work to do to fix those memory leaks 
?? thanks 

Original comment by amanriqu...@gmail.com on 17 Jan 2012 at 5:05

[GoogleCodeExporter]

Tracking down and fixing memory issues is very hard and takes a lot of time. 
I've spent a huge amount of time trying to detect and fix memory leaks in 
PowerMock but I suppose I haven't found all of them. I don't have time to track 
this down myself so I really would need help from the community to fix the 
remaining issues.

Original comment by johan.ha...@gmail.com on 17 Jan 2012 at 8:15

[GoogleCodeExporter]

Just a comment:

I was experiencing the same problem (with PowerMock 1.4.12), with PermGenSpace 
errors when building with Maven(2). I monitored my build with VisualVM, and 
noticed that Maven was far from running out of PermGenSpace. So, i examined the 
error message a bit closer, and noticed that it was SureFire that threw the 
error. After a bit of googleing this error message, I found that the problem 
was related to Maven's thread management, so I built with "mvn clean install 
-DforkMode=never". Now, I did not get a PermGenSpace error, but instead some of 
the tests failed (they passed when running in Intellij IDEA.)

I looked into the surefire report, and saw that something was wrong with the 
way something was mocked (I will get back to this), so i refactored the test, 
and ran "mvn clean install -DforkMode=never" again. The project built 
successfully. Then i tried running "mvn clean install" (without the forkMode 
parameter). This also failed with PermGenSpace, but after a different testclass 
than the original error.

So, the structure of one my tests was actually the culprit, combined with 
Maven/Surefires fork settings.

So, the first error first:

In the project I am working on, there is a structure where domain classes come 
in two versions, one annotated with JPA-annotations, used with Hibernate, and 
one for use elsewhere in the system (don't ask me why..). In between there are 
converter classes, which basically writes the contents of the variables of one 
class over to the other (the are basically identical). These converters are 
also static factories of themselves, so one gains an instance by calling 
***Converter.convertWithAllRelations(). Then, .convert(*** from) is called on 
this instance (where *** = the class in question).

When testing this, I tried the following (maybe not that thought through):

//set up the testclass with testrunner and preparefortest

        final ***Converter converter = mock(***Converter.class);

       PowerMockito.mockStatic(***Converter.class);
       PowerMockito.when(***Converterer.converWithReducedRelations()).thenReturn(converter);
        when(converter.convert(***from)).thenReturn(***to);

So, basically a static mock of the factory method, telling it to return a 
mocked instance when called.

This worked in IDEA, but with Maven, not too good.

Now, on to the second testclass that made a PermGenSpace error:

This turned out to be a bit trickier. I first tried commenting out the tests i 
added, leaving the class as it was, only with the PowerMockRunner testrunner 
and a preparefortest annotation (which I also added). This too resulted in a 
PermGenSpace error. So the problem was in one of the existing tests. So, I 
added @Ignore to all tests, and no PermGenSpace error was thrown. I then 
continued to unignore one test at the time, to try and figure out which test(s) 
lead to the error. Having any one of the tests unignored spawned the error, so 
I tried with a dummy test (assertTrue(true)), and the same happened. So it was 
something in my @Before-method. With the contents of the @Before method all 
commented out, my dummy-test ran without error. So, I narrowed it down line by 
line, and found that it was one of the mock(Something.class) statements that 
caused the error. 

The class to be mocked differed from most of the other classes in one aspect: 
It implemented Comparable. So, I made a dummy class that did the same, and 
tried to mock that, and quite surely, PermGenSpace error (tried using both 
Mockito and PowerMockito's mock method). The problem seems related to the 
@PrepareForTest-annotation, it worked with only the 
@RunWith(PowerMockRunner.class), and failed when PrepareForTest was added 
(tried both with and without the class to be mocked in the list of classes to 
be prepared.)

This turned into quite a long comment, but hopefully it will be of some help to 
others experiencing similar problems. To sum it up:

1. If your tests run fine in your IDE, but you get memory errors when running 
with Maven, try running it with -DforkMode=never, as it might uncover the true 
error. Also, investigate the test being run right before the error occurs, as 
it is probably the culprit.
2. Mocking both static and instance parts of the same class is troublesome with 
PowerMock (or maybe I just did it wrong).
3. Mocking classes that implement Comparable does not work when using the 
@PrepareForTest-annotation(this should perhaps be an issue itself).

Original comment by torbjor...@gmail.com on 14 May 2012 at 4:18

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

After some more trial and error:

It seems that I was wrong about the problem with Comparable, it gave a 
PermGenSpace error no matter what i mocked (other than util-classes). I guess 
this is because it really was low on PermGenSpace.

It seems to me that PowerMock has higher demands for PermGenSpace than Surefire 
delivers as standard (64m isn't it?). So the solution, that has been working so 
far at least, was to give the plugin some more memory by adding a line in the 
pom.xml:

      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <argLine>-XX:PermSize=128m -XX:MaxPermSize=512m</argLine>
        </configuration>
      </plugin>

(If you have a multi-module project, with redefinition of the plugin, add it to 
the pom.xml closest to your tests.)

Original comment by torbjor...@gmail.com on 14 May 2012 at 5:15

[GoogleCodeExporter]

Thanks for sharing your comments. There could be a memory leak lurking around 
somewhere they can be very difficult to find.

Original comment by johan.ha...@gmail.com on 14 May 2012 at 5:18

[GoogleCodeExporter]

I assume that Surefire uses its own runner to execute junit tests?

Would it not make sense to somehow incorporate the fix from sprynter into the 
surefire setup?

Original comment by alwyn.sc...@gmail.com on 15 Jul 2012 at 2:15

[GoogleCodeExporter]

We've got the same problem with Powermock 1.5, so it doesn't seem to be fixed. 
Instead of setting MaxPermSize on the surefire plugin, you can set 
reuseForks=false, then it will start a new JVM for each test (ie with a fresh 
PermGen). Otherwise you need to keep increasing MaxPermSize as more unit tests 
are added. 

Unfortunately JaCoCo plugin (and Sonar) does not work with reuseForks=false set 
on the surefire plugin :-( 

Original comment by justinba...@gmail.com on 18 Jul 2013 at 7:53

[GoogleCodeExporter]

Have you tried the version in trunk? Please try to build the latest version of 
powermock from source (skip tests and javadoc generation to make it quicker) 
and try if it works.

Original comment by johan.ha...@gmail.com on 18 Jul 2013 at 11:09

[GoogleCodeExporter]

We are using Powermock 1.5.1, the issue still exists.

Original comment by guru.pra...@gmail.com on 1 Jan 2014 at 11:48

[GoogleCodeExporter]

We are also having this problem and we are using 1.5.2

Original comment by cbr...@infinio.com on 8 Jan 2014 at 10:19

[GoogleCodeExporter]

The issue I am experiencing is actually this:  
https://code.google.com/p/powermock/issues/detail?id=207

Which seems to be fixed in the upcoming 1.6 release.  

Original comment by cbr...@infinio.com on 8 Jan 2014 at 10:35