search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

OutOfMemoryError (heap space) generating testng report #291

Closed mondrejko closed 11 years ago

mondrejko commented 12 years ago

Seeing the following during our test run (about 4100 tests). The issue does not occur consistently, but does occur regularly. Increasing the max heap size does not address issue (have gone from 1g to 2g to 4g with same failure). I was able to capture a heap dump; however, where should this be submitted?

[testng] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space [testng] at java.util.Arrays.copyOf(Arrays.java:2882) [testng] at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) [testng] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) [testng] at java.lang.StringBuffer.append(StringBuffer.java:224) [testng] at java.lang.StringBuffer.append(StringBuffer.java:284) [testng] at java.util.regex.Matcher.appendReplacement(Matcher.java:746) [testng] at java.util.regex.Matcher.replaceAll(Matcher.java:813) [testng] at org.testng.reporters.XMLStringBuffer.toXML(XMLStringBuffer.java:327) [testng] at org.testng.reporters.XMLReporter.generateReport(XMLReporter.java:66) [testng] at org.testng.TestNG.generateReports(TestNG.java:1088) [testng] at org.testng.TestNG.run(TestNG.java:1047) [testng] at org.testng.TestNG.privateMain(TestNG.java:1337) [testng] at org.testng.TestNG.main(TestNG.java:1306)

cbeust commented 12 years ago

I'm not seeing any attachment, feel free to email it directly to me if github is stripping it.

mondrejko commented 12 years ago

The heap dump can be downloaded at https://www.yousendit.com/download/TEhWZ296VEhCMTZjZDhUQw

Let me know if you have any issues retrieving the file. Thanks.

mondrejko commented 12 years ago

Cedric, did you find any useful information in the heap dump? Let me know if there is any additional information that I can get for you. The OOME is occurring regularly on our production build server. Thanks.

cbeust commented 12 years ago

Yes, it looks like the problem is that the HTML reporter is generating the report string in memory, and that this string takes up more than 40Mb. This is quite staggering, how many tests are you running?

Cédric

On Thu, Oct 18, 2012 at 1:45 PM, mondrejko notifications@github.com wrote:

Cedric, did you find any useful information in the heap dump? Let me know if there is any additional information that I can get for you. The OOME is occurring regularly on our production build server. Thanks.

— Reply to this email directly or view it on GitHubhttps://github.com/cbeust/testng/issues/291#issuecomment-9580584.

mondrejko commented 12 years ago

This run is currently around 4100 tests. I have found that I get the OOME even when the max heap size is increased from 1gb to 4gb.

Is there additional information that I could provide? Is there a fix that we could try in our environment?

cbeust commented 12 years ago

I think the problem is in org.testng.reporters.jq.Main#generateReporters: the variable xsb is a string buffer that accumulates the content of the entire report and it seems to become unreasonably big with your tests.

The first workaround is to disable this listener, which means you won't get the HTML reports (do you use them?).

If you need them, then we need to break xsb in smaller pieces. A good place to start would be the loop:

for (INavigatorPanel panel : panels) {
  panel.generate(xsb);
}

Instead of generating this in xsb, generate it in a new buffer each time and save that content to a file. Then at the end of the reporter, create the file by concatenating these files and xsb.

Does this make sense? Can you try this by patching TestNG yourself?

Cédric

On Fri, Oct 19, 2012 at 5:20 AM, mondrejko notifications@github.com wrote:

This run is currently around 4100 tests. I have found that I get the OOME even when the max heap size is increased from 1gb to 4gb.

Is there additional information that I could provide? Is there a fix that we could try in our environment?

— Reply to this email directly or view it on GitHubhttps://github.com/cbeust/testng/issues/291#issuecomment-9599088.

mqbik commented 12 years ago

We see similar issue in our project which right now have "only" around 180 tests. Runing suite results in OOME. Expanding heap space doesn't help. Could we do something with it?

We use testng 6.8

removing org.testng.reporters.XMLReporter listener fixes the problem and mvn task runs without errors

Stack trace:

org.apache.maven.surefire.util.SurefireReflectionException: java.lang.reflect.InvocationTargetException; nested exception is java.lang.reflect.InvocationTargetException: null
java.lang.reflect.InvocationTargetException
 org.apache.maven.surefire.util.SurefireReflectionException: java.lang.reflect.InvocationTargetException; nested exception is java.lang.reflect.InvocationTargetException: null
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:601)
  at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
  at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
  at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
  at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
  at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
Caused by: java.lang.OutOfMemoryError: Java heap space
  at java.util.Arrays.copyOf(Arrays.java:2367)
  at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
  at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
  at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
  at java.lang.StringBuffer.append(StringBuffer.java:237)
  at org.testng.reporters.XMLStringBuffer.addCDATA(XMLStringBuffer.java:308)
  at org.testng.reporters.XMLReporter.writeReporterOutput(XMLReporter.java:76)
  at org.testng.reporters.XMLReporter.generateReport(XMLReporter.java:61)
  at org.testng.TestNG.generateReports(TestNG.java:1089)
  at org.testng.TestNG.run(TestNG.java:1048)
  at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:122)
  at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:92)
  at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:101)
  ... 9 more
cbeust commented 12 years ago

It looks like it's not just the HTML reporter that's memory hungry but XMLReporter as well. Interesting that these two problems would pop up around the same time, for different users and for different reporters.

I guess I need to start thinking about creating these reports into their final files incrementally instead of creating them in memory and then saving them. I'll add this to my TODO list.

atomicknight commented 12 years ago

You may want to consider something similar to the StAX API (Java 6+): http://docs.oracle.com/javase/6/docs/api/javax/xml/stream/XMLStreamWriter.html

cbeust commented 12 years ago

I just uploaded a beta version of TestNG that should fix this problem, can you please try it at http://testng.org/beta ant report back?

If it doesn't work, please make sure you are using that jar file and not your current one, the banner should display:

... ... TestNG 6.9beta 20121109_2221 by C�dric Beust (cedric@beust.com) ...

cbeust commented 12 years ago

@atomicknight I chose a much more lightweight solution:

https://github.com/cbeust/testng/blob/master/src/main/java/org/testng/reporters/FileStringBuffer.java

atomicknight commented 12 years ago

@cbeust I don't think that implementation solves the problem. When FileStringBuffer#toString is invoked, it still starts out with a StringBuilder with the default capacity, meaning that it effectively triggers the same logic path that caused the OOME in the first place. This approach may work if you determine the size of the file first and then create the StringBuilder with an appropriate capacity, though I think it'd be quite a bit more efficient to incrementally write to the final file as you suggested previously. This alternate approach also allows for much larger reports to be generated, as there is no requirement to keep the entire report in memory at any time (given that most operations are appends rather than arbitrary inserts).

The XMLStringBuffer class is conceptually almost identical to the StAX XMLStreamWriter class except that the former writes to a memory buffer whereas the latter writes to an arbitrary Writer/OutputStream. Seems like this would be a relatively straightforward change to make.

cbeust commented 12 years ago

FileStringBuffer will never hold more than 100k at a time. The only thing that this class doesn't handle is you try to add one big string which I don't think any of the reporters ever do (it's easy to handle this situation if it turns out to be a problem).

What am I missing?

atomicknight commented 12 years ago

As I understand it, FileStringBuffer will never hold more than 100K characters in memory while building. However, this is not the case with FileStringBuffer#toString, which loads the entire buffer (including the overflow) into memory in a manner similar to the original logic.

Here's the call sequence that I believe to be problematic:

XMLReporter#generateReport calls XMLStringBuffer#toXML at line 66 XMLStringBuffer#toXML calls FileStringBuffer#toString at line 327 FileStringBuffer#toString calls Files#readFile at line 82

Within Files#readFile, a new StringBuilder is created. After reading 16 characters, the #expandCapacity method is called on the StringBuilder to resize it to 32, temporarily taking up 48 characters worth of space. After reading 16 more characters, the #expandCapacity method is called on the StringBuilder to resize it to 64, temporarily taking up 96 characters worth of space. Continuing on, you end up with the original situation in which #expandCapacity tries to allocate a new array that is twice the current size, but fails because the original array is already enormous.

Does this sound right?

cbeust commented 12 years ago

Good catch, Abraham, you're right: my current solution doesn't solve the problem. I'll fix it so that reporters write their XMLStringBuilders directly into the file by chunks.

Unfortunately, adding dependencies to libraries is something I need to avoid doing in TestNG.

cbeust commented 12 years ago

I just uploaded a new beta that fixes the problem pointed out by atomicknight, the new version should read:

... TestNG 6.9beta 20121111_2208

mqbik commented 12 years ago

Cedirc, Abraham, thanks for quick response. I just become father and I'm short of time. Will check it as soon as possible.

cbeust commented 12 years ago

@mkubik8080 As a father of two young twins myself, I sympathize :-)

mflahe commented 11 years ago

Hi all, I think I am running into this as well. I am running a very long script with tons of output. We tried running the latest TestNG we could find, 6.8.6beta.jar (5/25/2013 10:07 AM) and were still able to get an out of memory exception and trace (included at end of post.)

Do you know if this fix is present in the 6.8.6 beta? Or do I need to find "TestNG 6.9beta 20121111_2208" to get these changes running? I want to determine or eliminate the XML string as the cause.

Any input is greatly appreciated. Thanks in advance!

[testng] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space [testng] at java.util.Arrays.copyOf(Arrays.java:2367) [testng] at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) [testng] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) [testng] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) [testng] at java.lang.StringBuffer.append(StringBuffer.java:237) [testng] at org.testng.reporters.XMLStringBuffer.addCDATA(XMLStringBuffer.java:308) [testng] at org.testng.reporters.XMLReporter.writeReporterOutput(XMLReporter.java:76) [testng] at org.testng.reporters.XMLReporter.generateReport(XMLReporter.java:61) [testng] at org.testng.TestNG.generateReports(TestNG.java:1089) [testng] at org.testng.TestNG.run(TestNG.java:1048) [testng] at org.testng.TestNG.privateMain(TestNG.java:1338) [testng] at org.testng.TestNG.main(TestNG.java:1307)

atomicknight commented 11 years ago

The stack trace indicates that you're using an older version of TestNG. The last version of XMLStringBuffer that used a StringBuffer internally was 6.8; starting with 6.8.1, FileStringBuffer is used instead.

You should check to see whether you have another TestNG JAR on your classpath.

mflahe commented 11 years ago

Thank you, there was indeed an older version in the path. Turns out that selenium-server-standalone.jar can have a copy of TestNG in there.

Figured it out after removing all references to TestNG*.jar and STILL getting the offending stack trace.

Many thanks!

cbeust commented 11 years ago

Glad to hear your problem is solved, closing the issue.