search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.99k stars 1.02k forks source link

Parallel test (parallel='tests') does not work when priority is used in Test #1336

Closed perwez123 closed 7 years ago

perwez123 commented 7 years ago

TestNG Version :6.9.12

Note: only the latest version is supported

Expected behavior

When priority is used for all test case in all test class and test are run in parallel using parallel="tests" mode from testng.xml file.

Then all the test case must run in the same order as priority set for each test case without any issue

Actual behavior

There are 3 test class and each test class having 3 tests method. And all test methods having the priority starts from 1 to 3 in each test class. The last test cases (3 rd one) of each class supposed to navigate to the google URL that is given in the test cases

When test are from testNG.Xml file using parallel='"tests" then it does not execute all the test case from all class.Only first test case of each test class gets executed and remaining get skipped. But when the same test class has been executed using parallel="classes" then it worked fine.

Note:

If the priority are set in sequence then it works fine. For example 1-3 in first test class and 4-6 in second test class and 6-9 in third class . But issue occurs only when all test class having the same priority order.

It shows all test is passed from eclipse console, which incorrect as my test is expected to navigate to the url but it did not.To see the issue you need to look on to the each browser instance.

Is the issue reproducible on runner?

Test case sample

Please, share the test case (as small as possible) which shows the issue Steps to reproduce

Please copy all 4 class file which is mentioned below and create the maven project in eclipse and run the test from testng.xml

1.Run the test from Testng.xml file from eclipse 2.Now see the browser of all 3 instance.The 3 rd test case is supposed to navigate to the url(google), but it executed the first test case of first 2 test class and all test case of 3 rd test class

  1. It means it always execute only first test case of all test class and all test case of last test class.

But if you see the eclipse console then it will show all test is passed because no assertion was done in the test,It was just to navigate.

When priority is removed and ran the tests then it worked fine . So I believe there is an issue with priority and parallel="tests".

Important

Please use the testNG.xml file which I have provided to reproduce the issue. Do not change anything in this.

BrowserInit class, This class launch the browser using ThreadLocal.(You must use this class to launch the browser)

public class BrowserInit {
    private static ThreadLocal<WebDriver> driver = new ThreadLocal<WebDriver>();
    @BeforeClass
    public void beforeCLass(){
                  System.setProperty("webdriver.chrome.driver","/browserdriver/chrome/mac/chromedriver");
        driver.set(new ChromeDriver());
    }

    public WebDriver getDriver(){

        return driver.get();
    }

TestClass 1(TestNG1)

import org.testng.annotations.Test;
public class TestNG1 extends BrowserInit{

    @Test(priority=1)
    public void test1(){
        getDriver().navigate().to("http://testng.org/doc/download.html");

        System.out.println("TestNG Test 1");
    }

    @Test(priority=2)
    public void test2(){
        getDriver().navigate().to("https://github.com/pmd/pmd-eclipse-plugin/blob/master/ReleaseNotes.md");
        System.out.println("TestNG Test 2");
    }

    @Test(priority=3)
    public void test3(){
        getDriver().get("https://www.google.co.in/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=parallel%20testing%20using%20test%20ng");

        System.out.println("TestNG Test 3");
    }
}

TestClass 2(TestNG2)

public class TestNG2 extends BrowserInit
{

    @Test(priority=1)
    public void test1TestNG2(){
            getDriver().navigate().to("http://testng.org/doc/download.html");

        System.out.println("TestNG 2 Test 1");
    }

    @Test(priority=2)
    public void test2TestNG2(){
        System.out.println("TestNG2 Test 2");
        getDriver().navigate().to("http://www.nakov.com/inetjava/lectures/part-1-sockets/InetJava-1.3-Multithreading.html");

    }

    @Test(priority=3)
    public void test3TestNG2(){
        getDriver().get("https://www.google.co.in/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=parallel%20testing%20using%20test%20ng");

        System.out.println("TestNG2 Test 3");
    }
}

TestClass 3(TestNG3)

public class TestNG3 extends BrowserInit
{

    @Test(priority=1)
    public void test1TestNG3(){
        getDriver().navigate().to("http://testng.org/doc/download.html");

        System.out.println("TestNG3 Test 1");
    }

    @Test(priority=2)
    public void test2TestNG3(){
        getDriver().navigate().to("https://www.tutorialspoint.com/java/java_thread_synchronization.htm");

        System.out.println("TestNG3 Test 2");
    }

    @Test(priority=3)
    public void test3TestNG3(){
        getDriver().get("https://www.google.co.in/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=parallel%20testing%20using%20test%20ng");   

        System.out.println("TestNG3 Test 3");
    }
}

TestNG.xml file (You must use this file to reproduce the issue)

 <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Extent Parallel Test" parallel="tests" thread-count="6">

     <test name="Mac_Chrome">
        <classes>
            <class name="com.test.TestNG1" />
            <class name="com.test.TestNG2" />
            <class name="com.test.TestNG3" />

        </classes>
    </test>
</suite>
juherr commented 7 years ago

Thanks for the report. Do you think it is similar to #1066?

perwez123 commented 7 years ago

I think it is some what similar to this but not exactly. As in my case I have used parallel="tests" and have put all the test class in one test tag as shown below. But if i create test tag for each class separately then it works fine. And also it works fine when priority is removed from tests.

I think this issue exist in thread pooling. Thread is overlapped to last session after running the first test case of each test class.

Issue occurs when test runs in parallel as shown below

`<?xml version="1.0" encoding="UTF-8"?>`
`<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">`
`<suite name="Extent Parallel Test" parallel="tests" thread-count="6">`
    ` <test name="Mac_Chrome">`
          ` <classes>`
             ` <class name="com.test.TestNG1" />`
              `<class name="com.test.TestNG2" />`
            `  <class name="com.test.TestNG3" />`
           ` </classes>`
    `</test>`
`</suite>`

But the same tests works fine when it is executed in this way as shown below

`<?xml version="1.0" encoding="UTF-8"?>`
`<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">`
`<suite name="Extent Parallel Test" parallel="tests" thread-count="6">`
    `<test name="Mac_Chrome1">`
        `<classes>`
          ` <class name="com.test.TestNG1" />`
         `</classes>`
  `  </test>`

  `<test name="Mac_Chrome2"`
        `<classes>`
            `<class name="com.test.TestNG2" />`      
        `</classes>`    
    `</test>`
     `<test name="Mac_Chrome3">`
        `<classes>`
            `<class name="com.test.TestNG3" /> `         
        `</classes>`
    `</test>`  
`</suite>`

Note :

After making fixes for this issue and the other which you tagged , It must be tested using all possible scenario (Parallel tests) . Please let me know once these issues are fixed so that I can perform one round of testing.

Are you able to reproduce this issue using my code mentioned above ?

krmahadevan commented 7 years ago

@juherr - I think I can recreate the problem. I also have a hunch of what might be going wrong. I think its related to our GraphThreadPoolExecutor which picks up freenodes for execution. But the issue here is that it keeps submitting new tasks from the ready list, after a method has completely run to completion. This is very evident in cases, wherein a TestNG test is making use of Selenium wherein there are multiple test methods which share the same webdriver (the class that facilitates talking to a web browser). Here's a full fledged sample which if executed, will trigger the problem every single time and there will be test failures.

Pre-requistes :

Base Class

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;

public class BaseClass {
    private WebDriver driver;

    @BeforeClass
    public void beforeClass() {
        driver = new ChromeDriver();
    }

    WebDriver getDriver() {
        return driver;
    }

    @AfterClass
    public void afterClass() {
        driver.quit();
    }
}

Test class

public class TestNG1 extends BaseClass {
    @Test (priority = 1)
    public void test1TestNG1() throws InterruptedException {
        String url = "http://testng.org/doc/download.html";
        getDriver().get(url);
        Assert.assertEquals(getDriver().getCurrentUrl(), url);
    }

    @Test (priority = 2)
    public void test2TestNG1() throws InterruptedException {
        String url = "http://www3.lenovo.com/in/en/";
        getDriver().get(url);
        Assert.assertEquals(getDriver().getCurrentUrl(), url);
    }

    @Test (priority = 3)
    public void test3TestNG1() throws InterruptedException {
        String url = "https://github.com/";
        getDriver().get(url);
        Assert.assertEquals(getDriver().getCurrentUrl(), url);
    }

}

Suite xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="1336_Suite" verbose="2"  parallel="classes" thread-count="10">
    <test name="1336_Tests">
        <classes>
            <class name="com.rationaleemotions.github.issue1336.TestNG1"/>
            <class name="com.rationaleemotions.github.issue1336.TestNG2"/>
            <class name="com.rationaleemotions.github.issue1336.TestNG3"/>
        </classes>
    </test>
</suite>

The issue goes away if I resort to introducing a CountDownLatch in GraphThreadPoolExecutor as below. But I am not able to gauge the repercussions of this hack (I am already having trouble with getting the build to pass tests, which is kind of expected)

  private void runNodes(List<T> freeNodes) {
    List<IWorker<T>> runnables = m_factory.createWorkers(freeNodes);
    CountDownLatch latch = null;
    if (! runnables.isEmpty()) {
      latch = new CountDownLatch(runnables.size());
    }
    for (IWorker<T> r : runnables) {
      if (latch != null && (r instanceof TestMethodWorker)) { //Use latch only for testmethod workers
        r.setLatch(latch);
      }
      m_activeRunnables.add(r);
      ppp("Added to active runnable");
      setStatus(r, Status.RUNNING);
      ppp("Executing: " + r);
      try {
        execute(r);
      }
      catch(Exception ex) {
        ex.printStackTrace();
      }
    }
    if (latch != null) {
      try {
        latch.await();
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
    }
  }
perwez123 commented 7 years ago

Sorry for late update , This is can be reproduced without parallel test also following the same steps as mentioned above.

ryudice commented 6 years ago

Hi, I’m having the same issue with parallel=‘classes’, if you add a priority it will cause each of the test methods in a class to run in different threads. Would this fix that? And is this available in a release?

Thanks

krmahadevan commented 6 years ago

@ryudice - It should be part of released TestNG. What happens when you try using latest released version viz., 6.14.3 ? If you are still having a problem, I would request you to please create a new issue and include a full fledged simple standalone test, which we can use to reproduce the problem.

ryudice commented 6 years ago

hi @krmahadevan , I created #1773. The issue does happen with the latest release.

Arohk commented 5 years ago

I am getting the same issue with 6.14.3. When removing priority the tests run in parallel.

krmahadevan commented 5 years ago

The latest released version is 7.0.0-beta3. Please retry using this and report back the results.

OTSAppsJenkins commented 5 years ago

I am getting the same issue with 6.14.3.

parallel="tests" are not working.

org.openqa.selenium.WebDriverException: Method has not yet been implemented (WARNING: The server did not provide any stacktrace information) Command duration or timeout: 0 milliseconds Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03' System info: host: 'Vinoths-MacBook-Pro-4.local', ip: 'fe80:0:0:0:1069:571c:3483:79e8%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.5', java.version: '1.8.0_191' Driver info: io.appium.java_client.AppiumDriver Capabilities {browserName: chrome, chromeOptions: {androidPackage: com.android.chrome}, databaseEnabled: false, desired: {browserName: chrome, chromeOptions: {androidPackage: com.android.chrome}, deviceName: Android Emulator, eventTimings: true, maxTypingFrequency: 8, nativeWebTap: true, newCommandTimeout: 0, noReset: true, noSign: true, orientation: PORTRAIT, platform: ANDROID, platformName: Android, platformVersion: 8.0, proxy: {proxyAutoconfigUrl: http://127.0.0.1:19876/pac.js, proxyType: PAC}, udid: emulator-5554, unicodeKeyboard: true, version: , webdriver.remote.quietExceptions: false}, deviceManufacturer: unknown, deviceModel: Android SDK built for x86_64, deviceName: emulator-5554, deviceScreenSize: 768x1280, deviceUDID: emulator-5554, eventTimings: true, hasMetadata: true, javascriptEnabled: true, locationContextEnabled: false, maxTypingFrequency: 8, nativeWebTap: true, networkConnectionEnabled: true, newCommandTimeout: 0, noReset: true, noSign: true, orientation: PORTRAIT, platform: ANDROID, platformName: ANDROID, platformVersion: 8.0.0, proxy: Proxy(pac: http://127.0.0.1..., takesScreenshot: true, udid: emulator-5554, unicodeKeyboard: true, version: , warnings: {}, webStorageEnabled: false, webdriver.remote.quietExceptions: false, webdriver.remote.sessionid: 0cfc524ea0fe4d338dd79aa398f...} Session ID: 0cfc524ea0fe4d338dd79aa398f09ad8 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214) at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166) at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40) at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80) at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44) at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158) at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:231) at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552) at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:46) at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1) at org.openqa.selenium.remote.RemoteWebDriver.getCurrentUrl(RemoteWebDriver.java:287) at nbc_MobWeb_testcases.TC001_Top_stories_playback_with_pre_roll.Mob_NbcPage(TC001_Top_stories_playback_with_pre_roll.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.Invoker.invokeMethod(Invoker.java:583) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.privateRun(TestRunner.java:648) at org.testng.TestRunner.run(TestRunner.java:505) at org.testng.SuiteRunner.runTest(SuiteRunner.java:455) at org.testng.SuiteRunner.access$000(SuiteRunner.java:40) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:489) at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

krmahadevan commented 5 years ago

@OTSAppsJenkins - Please try using latest released version of TestNG 7.0.0-beta5 at the moment.

DarkoShishkaa commented 1 year ago

I am getting this issue on the newer versions of the TestNG 7.8.0 and 7.7.0. I have two devices that I want to test simultaneously and have two different tests for it in the .xml file

`<?xml version="1.0" encoding="UTF-8"?>`
`<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">`
`<suite name="Extent Parallel Test" parallel="tests" thread-count="6">`
    ` <test name="Test1">`
          ` <classes>`
             ` <class name="com.test.TestNG1"/>`
              `<class name="com.test.TestNG2"/>`
            `  <class name="com.test.TestNG3"/>`
           ` </classes>`
    `</test>`

    ` <test name="Test2">`
          ` <classes>`
             ` <class name="com.test.TestNG1"/>`
              `<class name="com.test.TestNG2"/>`
            `  <class name="com.test.TestNG3"/>`
           ` </classes>`
    `</test>`

`</suite>`

It looks something like this (I copied the code of the kind Sir at the top and changed it a bit)

In each class I have tests with priority 1, 2 and 3. Based on how I understand it, it should run class TestNG1 and all methods in it based on the priority, then it runs TestNG2 and all methods in it based on the priority and same process for the TestNG3. But instead it runs all methods with the priority 1 first then methods with priority 2 and at the end methods with priority 3, even though that they are in different classes. When I stop using parallel="tests", it is working correctly, it is executing classes in order.

juherr commented 1 year ago

@DarkoShishkaa The priority is suite scoped and the behavior you see is the expected one. The feature that is used for the behavior you expect is the dependsOnMethods property.