search

testng-team / testng

TestNG testing framework
https://testng.org
Apache License 2.0
1.98k stars 1.02k forks source link

Unable to run tests in parallel using TestNG, gradle and android studio #1657

Closed ppatellz closed 6 years ago

ppatellz commented 6 years ago

TestNG Version

Note: only the latest version is supported

6.13.1

Expected behavior

The tests should run in parallel with following testng options in gradle task

useTestNG() {
                options {
                    suites 'src/test/java/com/android/lztestinglibrary/DeviceTesting/Appium/testng.xml'
                    setParallel('tests')
                    setThreadCount(2)
                }

Actual behavior

The first test runs twice (duplicate) in parallel in 2 threads. After that the remaining tests run in serial manner in a single thread.

Is the issue reproductible on runner?

Test case sample

Please, share the test case (as small as possible) which shows the issue

I posted my sample test case on: https://stackoverflow.com/questions/48015143/unable-to-run-tests-in-parallel-using-testng-gradle-and-android-studio

krmahadevan commented 6 years ago

What happens when you take out Gradle out of the equation and just run the TestNG tests via the TestnG command line ?

ppatellz commented 6 years ago

When I build the project using android studio, the test classes under src/test/java are not getting compiled. I tried instructions at: https://stackoverflow.com/questions/17894938/getting-error-could-not-find-or-load-main-class-org-testng-testng When I try to run command - cd C:\Users\ppatel

java -cp C:\Users\ppatel.gradle\caches\modules-2\files-2.1*; C:\Users\ppatel\LZTestingFramework\lztestinglibrary\build\intermediates\classes\debug org.testng.TestNG testng.xml

I get error message: Error: Could not find or load main class C:\Users\ppatel\LZTestingFramework\lztestinglibrar y\build\intermediates\classes\debug

Basically, my testng jar is inside files-2.1* and the compiled .class files are under folders under debug. However the debug folder does not have .class files for all my test .java files under src/test/java. Looks like it only compiled .java files under src/main/java.

Android studio structure is new to me and I am stuck here. Ideally we would like to use a gradle command to build the project and run the tests.

kool79 commented 6 years ago

Cannot add more detail now but want to report that issue is reproduced with v6.13.1 + maven-surefire runner (v2.20.1): All tests (except those with DataProvider(parallel=true)) are executed in the same "main" thread. Rollback to v6.10 fixes issue (I did not try other versions between 6.10 and 6.13.1)

textng.xml:

<suite name="Maven" verbose="3" configfailurepolicy="continue" parallel="classes" data-provider-thread-count="4" thread-count="8">
<listeners>
   <listener class-name="qa.ok.projects.shared.testng.externalsuite.TestSuiteTransformer"/>
</listeners>
<test name="Preconditions">
<packages>
   <package name="qa.ok.projects.testpreconditions.*"/>
</packages>
</test>
<test name="Main">
   .....

pom.xml:

<plugin>

  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.20.1</version>
  <dependencies>
    <dependency>
      <groupId>org.apache.maven.surefire</groupId>
      <artifactId>surefire-testng</artifactId>
      <version>2.20.1</version>
    </dependency>
  </dependencies>

  <configuration>
    <testSourceDirectory>${basedir}/src/main/java/</testSourceDirectory>
    <testClassesDirectory>${project.build.directory}/classes/</testClassesDirectory>
    <reportsDirectory>./test-output</reportsDirectory>

    <suiteXmlFiles>
      <suiteXmlFile>suites/testng.xml</suiteXmlFile>
    </suiteXmlFiles>

    <properties>
      <property>
        <name>configfailurepolicy</name>
        <value>continue</value>
      </property>
      <property>
        <name>dataproviderthreadcount</name>
        <value>10</value>
      </property>
      <property>
        <name>threadcount</name>
        <value>11</value>
      </property>
      <property>
        <name>surefire.testng.verbose</name>
        <!-- Set to -1 to put testng to debug mode-->
        <value>1</value>
      </property>
    </properties>

    <systemPropertyVariables>
      ....
    </systemPropertyVariables>
    <argLine>-Djava.util.logging.config.file=''</argLine>
  </configuration>
</plugin>

Notes: 1) TestSuiteTransformer does not change any parameters which are related to thread count (it just marks some tests as 'disabled' and replace data-providers) 2) I know there are some properties in pom.xml (<confirguration>/<pom>) which don't not have affect to testng properties (values in testng.xml have higher priority) but they are exists in my pom file (I just put it here 'as is').

kool79 commented 6 years ago

@krmahadevan, I created simple sample and tried to run it via command-line. ( test-parallel.zip ) Tests prints to console method name and Thread.currentThread().toString() Issue is reproduced for v6.13 and 6.13.1 and DO NOT reproduced for v6.11

Unfortunately v6.13 (as same as v6.11) don't generate verbose output. I tested the same settings with v6.9.10 - it has much more log-lines. Is it another issue?

output for v6.11: (parallel mode - OK)

D:\dev\AC\git\test-testng\target\test-classes>java -Dtestng.test.classpath="D:\dev\AC\git\test-testng\target\test-classes" -cp "D:\app\bin\testng-6.11.jar;D:\app\bin\jcommander-1.7
2.jar;D:\dev\AC\git\test-testng\target\test-classes" org.testng.TestNG D:\dev\AC\git\test-testng\target\test-classes\parallel-test.xml
...
... TestNG 6.11 by C?dric Beust (cedric@beust.com)
...

I'm test1FromClass1 in thread: Thread[TestNG-test=MyTest-1,5,main]
I'm test1FromClass2 in thread: Thread[TestNG-test=MyTest-3,5,main]
I'm test2FromClass1 in thread: Thread[TestNG-test=MyTest-2,5,main]
===== Invoked methods
    TestNgParallelTestsClass2.test1FromClass2()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass2@46f5f779] 1190524793
    TestNgParallelTestsClass1.test1FromClass1()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass1@42f30e0a] 1123225098
    TestNgParallelTestsClass1.test2FromClass1()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass1@42f30e0a] 1123225098
=====
PASSED: test1FromClass2
PASSED: test1FromClass1
PASSED: test2FromClass1

===============================================
    MyTest
    Tests run: 3, Failures: 0, Skips: 0
===============================================

===============================================
MySuite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

output for v6.13.1 (parallel mode - DOES NOT work)

D:\dev\AC\git\test-testng\target\test-classes>java -Dtestng.test.classpath="D:\dev\AC\git\test-testng\target\test-classes" -cp "D:\app\bin\testng-6.13.1.jar;D:\app\bin\jcommander-1
.72.jar;D:\dev\AC\git\test-testng\target\test-classes" org.testng.TestNG D:\dev\AC\git\test-testng\target\test-classes\parallel-test.xml
...
... TestNG 6.13.1 by C?dric Beust (cedric@beust.com)
...

I'm test1FromClass1 in thread: Thread[main,5,main]
I'm test2FromClass1 in thread: Thread[main,5,main]
I'm test1FromClass2 in thread: Thread[main,5,main]
===== Invoked methods
    TestNgParallelTestsClass1.test1FromClass1()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass1@6979e8cb] 1769597131
    TestNgParallelTestsClass1.test2FromClass1()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass1@6979e8cb] 1769597131
    TestNgParallelTestsClass2.test1FromClass2()[pri:0, instance:qa.ok.parallel.TestNgParallelTestsClass2@2be94b0f] 736709391
=====
PASSED: test1FromClass1
PASSED: test2FromClass1
PASSED: test1FromClass2

===============================================
    MyTest
    Tests run: 3, Failures: 0, Skips: 0
===============================================

===============================================
MySuite
Total tests run: 3, Failures: 0, Skips: 0
===============================================
ppatellz commented 6 years ago

@kool79 thanks for trying. However my question was in regards to runner as Gradle and NOT maven ? With gradle, the parallelization does not work with testng v 6.13 or 6.11. or 6.10.

kool79 commented 6 years ago

@ppatellz, I just have the same issue in v13 so its logical add more info here instead of opening new similar (or maybe same) issue. Also krmahadevan asked for logs when run it via command line - I did this. BTW in your first topic there are nothing about v6.10 and 6.11, so I decided it is the same issue. Please, try one more time with 6.10, 6.11 and ensure it is testNG bug and not gradle.

nivimor commented 6 years ago

I encountered the same issue. I ran the tests directly from testng.xml, as well as running gradle test task. I made sure to format the task as:

test{
   useTestNG()
}

In both attempts TestNG 6.13 ran the tests sequentially. In the suite, I tried to run both tests and classes in parallel but it always ran sequentially.

I downgraded to 6.10 and parallel is currently working when running the tests directly from testng.xml, but not when running it using gradle (as mentioned above).

ppatellz commented 6 years ago

@nivimor I posted my gradle task code in my original post stackoverflow link - https://stackoverflow.com/questions/48015143/unable-to-run-tests-in-parallel-using-testng-gradle-and-android-studio Please explain the exact steps of doing the following you mentioned in your post - 'I downgraded to 6.10 and parallel is currently working when running the tests directly from testng.xml'. Did you mean you ran testng.xml from android studio ide using 'Run' menu at the top. For me running the tests from 'Run' menu does NOT run in parallel. It runs sequential only.

@kool79 I tried again using the testng 6.10 and 6.11 , it runs sequential only.

krmahadevan commented 6 years ago

@kool79 ,

Can you please check if this issue is related to https://github.com/cbeust/testng/issues/1636 I have fixed this in the current snapshot 6.14-SNAPSHOT.

nivimor commented 6 years ago

@ppatellz

I'm using TestNG in the context of Appium testing. I'm using intellij (if I'm not mistaken Android Studio is based on intellij) and when I run testng.xml from the menu (alt+shift+f10) or directly from the file (right clicking → run or ctrl+shift+f10), tests run in parallel.

However, 6.10 doesn't run in parallel if I execute it using gradle. My gradle test task doesn't specify parallel or thread count, I specified these in the testng.xml. I only specified the suites option to point to the testng.xml.

ppatellz commented 6 years ago

@nivimor Even though Android studio (I am using latest 3.0) is based on intellij, when I run testng.xml from menu or directly from the file (right clicking), the tests still run serially. Here is the testng.xml file that I tried with testng v 10 and 11.

<suite name="test suite" parallel="methods" thread-count="3">

    <test name="all-tests" >
        <packages>
            <package name="com.android.testinglibrary.Appium.Tests"/>
        </packages>
    </test>
</suite>

@krmahadevan Looks like this is gradle and android studio specific issue. What should I do ?

juherr commented 6 years ago

@ppatellz Maybe you can report the issue here https://developer.android.com/studio/report-bugs.html but I'm not sure TestNG is supported by Android Studio (even if it is an IntelliJ savor). Maybe you can try to reproduce the issue with IntelliJ + Android plugin and report the issue https://youtrack.jetbrains.com/issues/IDEA

kool79 commented 6 years ago

@krmahadevan,

Can you please check if this issue is related to#1636

Checked, parallel runs now works in 6.14-SNAPSHOT

ppatellz commented 6 years ago

@juherr Per your suggestion, I opened the bug for android studio team - https://issuetracker.google.com/issues/71646139 There is NO asignee to this issue yet.

krmahadevan commented 6 years ago

@juherr - I guess we can close this issue right since TestNG is no longer involved for @ppatellz problem and we have fixed the problem in 6.14-SNAPSHOT which @kool79 had called out. What do you suggest ?

juherr commented 6 years ago

@ppatellz For the moment, we suppose it is an Android Studio issue. Please reopen the issue if you have any new information.