Closed ajay516 closed 7 years ago
Here is the code: https://github.com/ajay516/ajay516-testng-execution-order
Yeah, i can reproduce that.
Probably workaround: use testng.xml with preserve-order="true".
After some investigations i have found that "dependOnMethods" matter. Without dependency all works fine. Looks like TestNG incorrectly reorders test methods to run.
Simplified code you can find at VladRassokhin/testng-execution-order
Is there a workaround (except removing all dependsOnMethods)?
any clue of the root cause? action plan? The reason (at least for my use case:selenium ui testing) to choose testng over junit is tests dependencies.
order all tests in xml is not feasible for large project with hundreds, thousands tests.
If you want precise ordering, you should use dependsOnGroups, not XML (which was never designed to specify ordering).
dependsOnGroups does not work, at least not as my understanding. with parallel="false", I expect suite runs all tests in single thread, by the order defined either java5 annotations or xml (with preserve-order). depended groups/tests should be executed only once.
given one base class, and two sub classes, depended group in base class
the expected output:
beforeSuite
init
testA1
testA2
testB1
testB2
actual output:
beforeSuite
init
init
testA1
testA2
testB1
testB2
===============================================
Custom suite
Total tests run: 6, Failures: 0, Skips: 0
===============================================
suite.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Custom suite" parallel="false">
<test name="Test dependencies">
<classes>
<class name="SubClassA"/>
<class name="SubClassB"/>
</classes>
</test>
</suite>
public class BaseClass {
@BeforeSuite
public void beforeSuite() {
System.out.println("beforeSuite");
}
@Test(groups = {"init"})
public void init(){
System.out.println("init");
}
}
public class SubClassA extends BaseClass {
@Test(dependsOnGroups = {"init.*"})
public void testA1() {
System.out.println("testA1");
}
@Test(dependsOnMethods = "testA1")
public void testA2() {
System.out.println("testA2");
}
}
public class SubClassB extends BaseClass{
@Test(dependsOnGroups = {"init.*"})
public void testB1() {
System.out.println("testB1");
}
@Test(dependsOnMethods = "testB1")
public void testB2(){
System.out.println("testB2");
}
}
tell me, if I understand the documentation in a wrong way. Appreciate, if anyone can tell me how to properly use test dependencies in testng.
The ordering looks correct to me. What you seem to be wanting is atomicity (invoking groups of methods together)?
how could it be right? per http://testng.org/doc/documentation-main.html#testng-xml "TestNG will run your tests in the order they are found in the XML file"
my understand is, TestNG executes tests in the order found in the XML file, within each test, annotations (dependsOnGroup, dependsOnMethods) will decide the order.
so, I have SubClassA before SubClassB in xml, I expect TestNG execute SubClassA before doing anything about SubClassB. no?
The depend annotations override anything that is found in the XML. Either you use preserve-order=true in your XML file or you define dependencies in Java.
Thanks for the clarification. One more question, when you say "override anything that is found in the XML", that also apply to "group-by-instances" / "order-by-instances"? There is no way to archive it by using annotations?
btw, should it be "group-by_instances" or "order-by-instances"? Neither of them exists in http://testng.org/testng-1.0.dtd
from http://testng.org/doc/documentation-main.html
For this ordering, you can use the XML attribute group-by-instances. This attribute is valid either on <suite> or <test>:
<suite name="Factory" order-by-instances="true">
or
<test name="Factory" order-by-instances="true">
Hello,
I am using the latest version of TestNG (6.8.8) and I don't know how to solve an issue that I have. I have on Jenkins a job that executes some automation tests, These are TestNG tests and they are part of a suite file (testNG.xml). The execution order is the one from the suite file, but when I look into the testng-result.xml and then in emailable-report.html, the tests are in a random order.
I would like the tests from the reports to be in the order that they have executed. Can you please help me in solving this issue?
Thank you
Hello,
I've created a pull request that, I believe, fixes this issue: https://github.com/cbeust/testng/pull/554
I just upgraded my project (from 6.0.1 to 6.8) and found that the order of execution of tests are changed. In version of 6.0.1, the order of execution is:
In version 6.8, the order becomes:
Here are my observations:
Here is my execution environment: