qmetry / qaf

Quality Automation Framework for web, mobileweb, mobile native and rest web-service using Selenium, webdrier, TestNG and Java Jersey
https://qmetry.github.io/qaf
MIT License
253 stars 137 forks source link

java.lang.OutOfMemoryError: Java heap space when generate report #329

Open MoonLikeSinging opened 4 years ago

MoonLikeSinging commented 4 years ago

QAF Version

2.3 6.10 1.2.5 2.1.14 2.1.14 1.20.1

Steps To Reproduce

1. use verifyThat to validate more that 200000 records

Expected behavior

Generation report successfully.

Actual behavior

java.lang.OutOfMemoryError: Java heap space Preparing For Shut Down... at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at com.qmetry.qaf.automation.core.HtmlCheckpointResultFormatter.getResults(HtmlCheckpointResultFormatter.java:54) at com.qmetry.qaf.automation.core.HtmlCheckpointResultFormatter.getResults(HtmlCheckpointResultFormatter.java:56) at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener.report(QAFTestNGListener.java:248) at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener2.report(QAFTestNGListener2.java:169) at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener.onTestFailure(QAFTestNGListener.java:121) at org.testng.internal.Invoker.runTestListeners(Invoker.java:1716) at org.testng.internal.Invoker.runTestListeners(Invoker.java:1699) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1236) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:781) at org.testng.TestRunner.run(TestRunner.java:635) at org.testng.SuiteRunner.runTest(SuiteRunner.java:387) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340) at org.testng.SuiteRunner.run(SuiteRunner.java:289) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293) at org.testng.TestNG.runSuitesLocally(TestNG.java:1218) at org.testng.TestNG.runSuites(TestNG.java:1133) at org.testng.TestNG.run(TestNG.java:1104) at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66) at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)

Is the issue reproducible on runner?

Yes, it can be reproduced everytime.

Test case sample

Please, share the test case (as small as possible) which shows the issue

cjayswal commented 4 years ago

did you tried disabling default testng listeners? If it didn't resolves issue Increase Available Memory

MoonLikeSinging commented 4 years ago

Hi @cjayswal, I use this listeners: com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory, and set vm : -Xms1024m -Xmx4096m -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=4096m

cjayswal commented 4 years ago

In your ant or maven you need to make sure default listeners are false. In Ant

 <testng useDefaultListeners=false ...

For maven

 <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M4</version>
        <configuration>
          <properties>
             <property>
              <name>usedefaultlisteners</name>
              <value>false</value> <!-- disabling default listeners is optional -->
            </property>
MoonLikeSinging commented 4 years ago

ok, let me try it, thanks

MoonLikeSinging commented 4 years ago

hi @cjayswal I have disabled default listeners and set vm -Xms1024m -Xmx8192m -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=8192m, now I get error qaf.automation.testng.report.MethodResult in file: test-results/星期一16三月20_0530下午/xxx/com.qmetry.qaf.automation.step.client.Scenario/xxx.json :Requested array size exceeds VM limit. After the test finish, the json file is empty.

cjayswal commented 4 years ago

I am curious to know what you are testing? If you don't mind will you share details of your test case requirement and how you are implementing it?

Is it the case where you are doing tooooo much of interactions in the test case and it generates huge command log which goes out of memory? If so, there are different ways to deal with it.

Option 1:

One of the simplest way is, add exclude commands from logging using property reporter.log.exclude.commands. It will reduce the command log depending on what commands you excluded. For example:

reporter.log.exclude.commands = findElement,findElements,findChildElement,findChildElements,clearElement,getPageSource

If you add above property, It should reduce command log by 30-50%. You can find other command names in org.openqa.selenium.remote.DriverCommand class or refer constant values in java doc to add more command for excluding in command log.

Option 2:

Another option is using your own command log reporter. That can be set using driver listener. Set your own reporter and handle command log your way:

@Override
public void onInitialize(QAFExtendedWebDriver driver){
   //this can be your custom implementation by extending  WebDriverCommandLogger
   WebDriverCommandLogger reporter = new WebDriverCommandLogger();
   driver.setReporter(reporter);
}
Option 3:

In worst case, add a driver listener and in before command clear the command logs. You will not see any command log except last few.

@Override
public void beforeCommand(QAFExtendedWebDriver driver, CommandTracker commandTracker){
   //clear log
   TestBaseProvider.instance().get().getLog().clear();
}
MoonLikeSinging commented 4 years ago

Hi @cjayswal, We are migration data use different db, and we validate each records in the table, and we use verifyThat to verify, so it generated large of logs, and it looks the json file is generated after the test finished, so get Requested array size exceeds VM limit. We decide use Logger to record the validation logs now. Thanks for you help.

cjayswal commented 4 years ago

Will you please try using propertyreport.log.skip.success=true with 3.0.0-SNAPSHOT? If you are not able to resolve you can add below repository entry in your pom.

    <repository>
      <id>nexus-snapshots</id>
      <name>OSS nexus-snapshots</name>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </repository>

It can be found here.

MoonLikeSinging commented 4 years ago

I will try it when I free, I use log file to save the validation log now. After test I wiil reply you,thanks

MoonLikeSinging commented 4 years ago

Hi @cjayswal , Actually I used a framwork named Quantum, it's base on qaf. After I update the 3.0.0-SNAPSHOT, and run the testng file, I meet an testngException: org.testng.TestNGException: The factory method class com.qmetry.qaf.automation.step.client.ScenarioFactory.getTestsFromFile() threw an exception at org.testng.internal.FactoryMethod.invoke(FactoryMethod.java:121) at org.testng.internal.TestNGClassFinder.(TestNGClassFinder.java:153) at org.testng.internal.TestNGClassFinder.(TestNGClassFinder.java:40) at org.testng.TestRunner.initMethods(TestRunner.java:403) at org.testng.TestRunner.init(TestRunner.java:252) at org.testng.TestRunner.init(TestRunner.java:222) at org.testng.TestRunner.(TestRunner.java:171) at com.qmetry.qaf.automation.testng.TestRunnerFactory.newTestRunner(TestRunnerFactory.java:55) at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:623) at org.testng.SuiteRunner.init(SuiteRunner.java:189) at org.testng.SuiteRunner.(SuiteRunner.java:136) at org.testng.TestNG.createSuiteRunner(TestNG.java:1375) at org.testng.TestNG.createSuiteRunners(TestNG.java:1355) at org.testng.TestNG.runSuitesLocally(TestNG.java:1209) at org.testng.TestNG.runSuites(TestNG.java:1133) at org.testng.TestNG.run(TestNG.java:1104) at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66) at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.FactoryMethod.invoke(FactoryMethod.java:89) ... 17 more Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/text/StrLookup at com.qmetry.qaf.automation.data.MetaDataScanner.getParameter(MetaDataScanner.java:129) at com.qmetry.qaf.automation.data.MetaDataScanner.getParameter(MetaDataScanner.java:151) at com.qmetry.qaf.automation.step.client.ScenarioFactory.getTestsFromFile(ScenarioFactory.java:69) ... 22 more Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.text.StrLookup at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 25 more

The Listeners it used class-name="com.quantum.listeners.QuantumReportiumListener" and .

I don't know how to deal with this exception. The quamtum url: https://github.com/Project-Quantum/Quantum-Starter-Kit/blob/master/pom.xml