search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.98k stars 1.02k forks source link

Verbose mode > 4 can crash with Maven failsafe #1069

Open manosnoam opened 8 years ago

manosnoam commented 8 years ago

This is weired. When running TestNG + Maven failesafe, with surefire.testng.verbose > 4:

<configuration>
    <suiteXmlFiles>
        <suiteXmlFile>My_Test_Suite.xml</suiteXmlFile>
    </suiteXmlFiles>
    <systemPropertyVariables>

        <surefire.security.manager>java.lang.SecurityManager</surefire.security.manager>
        <java.security.manager>default</java.security.manager>
    </systemPropertyVariables>

    <useSystemClassloader>false</useSystemClassloader>
    <threadCount>20</threadCount>
    <reuseForks>true</reuseForks>
    <argLine>-Xmx2048m</argLine>
    <properties>
        <property>
            <name>listener</name>
            <value>org.imaginea.jenkins.plugins.testinprogress.testng.TestNGProgressRunListener</value>
        </property>
        <property>
            <name>surefire.testng.verbose</name>
            <value>10</value>
        </property>
    </properties>
</configuration>

On Windows 2008 R2, all tests fail right away. If I change it to Verbose < 5, everything run as usual.

I'm not sure if the following exception is the cause:

Creating c:\jenkins-slave\workspace\Automation_Matrix_New\72b9012b\test-output\testng-failed.xml java.io.IOException: The filename, directory name, or volume label syntax is incorrect at java.io.WinNTFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at org.testng.internal.Utils.writeFile(Utils.java:173) at org.testng.internal.Utils.writeUtf8File(Utils.java:135) at org.testng.reporters.FailedReporter.generateFailureSuite(FailedReporter.java:76) at org.testng.reporters.FailedReporter.generateReport(FailedReporter.java:47) at org.testng.TestNG.generateReports(TestNG.java:1135) at org.testng.TestNG.run(TestNG.java:1081) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:281) at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:121) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)

manosnoam commented 8 years ago

Full output log: console.txt

juherr commented 8 years ago

Thank for the stacktrace.

But I think the problem comes from

java.lang.RuntimeException at org.testng.internal.TestResult.toString(TestResult.java:251) at org.testng.internal.TestResult.toString(TestResult.java:236) at org.testng.internal.Utils.toString(Utils.java:655) at org.testng.reporters.VerboseReporter.logTestResult(VerboseReporter.java:216) at org.testng.reporters.VerboseReporter.onConfigurationFailure(VerboseReporter.java:67) at org.testng.internal.Invoker.runConfigurationListeners(Invoker.java:1664) at org.testng.internal.Invoker.handleConfigurationFailure(Invoker.java:303) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:241) at org.testng.internal.Invoker.invokeMethod(Invoker.java:589) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112) at org.testng.TestRunner.privateRun(TestRunner.java:782) at org.testng.TestRunner.run(TestRunner.java:632) at org.testng.SuiteRunner.runTest(SuiteRunner.java:366) at org.testng.SuiteRunner.access$000(SuiteRunner.java:39) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:400) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)

But your exception is strange too and should not append.

@cbeust The exception on newFile sound familiar. If I remember well you had already the same issue months ago, did you remember if you fixed the problem?

cbeust commented 8 years ago

This error doesn't seem to be related to what I used to see (which is fixed in 6.9.11 as far as I can tell).

juherr commented 8 years ago

@manosnoam What TestNG version do you use?

manosnoam commented 8 years ago

6.9.10 is my testng version, is it fixed on 6.9.10 ?

Thanks!

manosnoam commented 8 years ago

On 6.9.11 I mean ?

juherr commented 8 years ago

Yes, it should be but 6.9.11 and later are only available on bintray for the moment, not maven central.

manosnoam commented 8 years ago

Hi @juherr, It doesn't seems to be resolved using TestNG 6.9.11:

Failed to invoke configuration method com.galilsoftware.AF.testConfig.BaseTest.renameDDTmethod:null
ERROR 2016-06-19 13:30:15,606 TC-10128:_Counters_-_Current_Status_-_Add_modify_delete_-_internet_explorer [TestNG] [com.galilsoftware.AF.listeners.TestNGListener.onTestFailure] *** Test has Failed:  TC-10128: Counters - Current Status - Add modify delete - null ***
ERROR 2016-06-19 13:30:15,612 TC-10128:_Counters_-_Current_Status_-_Add_modify_delete_-_internet_explorer [TestNG] [com.galilsoftware.AF.listeners.TestNGListener.onTestFailure] Exception on: 
java.lang.RuntimeException
    at org.testng.internal.TestResult.toString(TestResult.java:251)
    at org.testng.internal.TestResult.toString(TestResult.java:236)
    at org.testng.internal.Utils.toString(Utils.java:657)
    at org.testng.reporters.VerboseReporter.logTestResult(VerboseReporter.java:216)
    at org.testng.reporters.VerboseReporter.onConfigurationFailure(VerboseReporter.java:67)
    at org.testng.internal.Invoker.runConfigurationListeners(Invoker.java:1632)
    at org.testng.internal.Invoker.handleConfigurationFailure(Invoker.java:304)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:242)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:590)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:823)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:778)
    at org.testng.TestRunner.run(TestRunner.java:632)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
    at org.testng.SuiteRunner.access$000(SuiteRunner.java:39)
    at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:400)
    at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This doesn't happen if using surefire.testng.verbose = -1.

I don't know if it's relevant or not: renameDDTmethod() method is aimed to give a new method name, since I only have 1 Test Method, that runs each time with new parameters and method reflection (based on data from excel), so I need to change the TestNG result - method name accordingly:


@BeforeMethod(alwaysRun = true)
    public void renameDDTmethod(ITestResult result, Method testMethod, Object[] parameters) {
        String methodName = testMethod.getName();
        if (methodName.equals("createDataDrivenTests") && parameters.length == 2) {
            methodName = "Login";
            String methodNameDefinition = ddt.getValue("Excel_Method_Field_Definition", "Test_Name");
            if (methodNameDefinition != null) {
                @SuppressWarnings("unchecked")
                String newMethodName = ((Hashtable<String, String>) parameters[0]).get(methodNameDefinition);
                if (newMethodName != null)
                    methodName = newMethodName;
                result.setAttribute("ddtTestName", methodName);
            }

            String methodDescriptionDefinition = ddt.getValue("Excel_Description_Field_Definition", "Description");
            if (methodDescriptionDefinition != null) {
                @SuppressWarnings("unchecked")
                String methodDescription = ((Hashtable<String, String>) parameters[0]).get(methodDescriptionDefinition);
                if (methodDescription != null)
                    result.setAttribute("ddtTestDescription", methodDescription);
            }
        }
        result.setAttribute("ddtParams", ddt.getData());
        this.setTestName(methodName);

    }
manosnoam commented 8 years ago

Attaching Jenkins execution (both Maven Compile using TestNG 6.9.11 + Maven TestNG Execution): Maven_Compile_TestNG-6.9.11.txt Maven_Run_TestNG_Failure.txt

juherr commented 8 years ago

Ok, thank.

I had a look and the failing log is only activated with >4. I don't understand what is the problem for the moment but feel free to analyse it :)