Test method are interleaved at runtime when "dependsOnGroups" is used

[andrewah]

Hi,

my application's test suite is based on testng 6.9.10 & uses a combination of classes which variously do and do not groups tests together.

When the entire suite is executed, tests are executed out of order. Ie. some tests in one class may be executed, then some tests in another class may be executed, before the remainder of tests declared in the first class are executed. I have forked the testng repo (see https://github.com/andrewah/testng) & put a test case together to demonstrate this in action - the files that I have added/changed are:

src/test/java/test/interleaving/TestInterleaving.java src/test/java/test/interleaving/TestLegacyShredding.java src/test/java/test/interleaving/TestPurging.java src/test/resources/testng.xml

When the test is executed the tests are executed in the following order:

TestPurging/TestLoadRefData TestLegacyShredding/testFileWithEmptyBody TestLegacyShredding/testIncorrectHashValue TestLegacyShredding/testIncorrectRecordCount TestLegacyShredding/testSingleValidFile TestPurging/TestShredAfterPurgeDate TestPurging/TestShredEarlierThanPurgeDate TestPurging/TestShredEqualToPurgeDate TestPurging/TestShredRecordToError TestPurging/TestShredRecordToFix TestPurging/TestStandardiseData TestPurging/TestLoadExtraRefData TestPurging/TestStandardiseErrorRecord TestPurging/TestPurge

I was expecting for the all of the tests in the "TestPurging" class to have been executed in consecutive order, but unfortunately that is not happening.

It would be great if you could look at the unit test I've put together and let me know if there is anything that can be done.

Thanks,

Andrew.

[skirkpatrick]

This looks like the expected output to me. According to Cedric in this google group thread the default behavior of preserving test order is ignored if other ordering mechanisms are used, such as dependsOnMethods/dependsOnGroups.

If you want to ensure that tests within the same test class are not being split up, you can use TestNG.setGroupByInstances(true), or group-by-instances="true" if using a testng.xml.

I've found that you may also need to specify TestNG.setPreserveOrder(false) or preserve-order="false" if using group-by-instances, otherwise you could end up with a circular dependency in your test graph (#565).

[andrewah]

Thanks for the response, I'll take a look at the options that you've presented. We don't use a testng.xml - we just invoke our tests via gradle using either the "test" task (to run all tasks) or via the "--tests" switch (ti invoke a single, or a subset of the test suite).

It's disappointing to hear that this is the expected behaviour. To echo the sentiments of many commentators to #106 , the fact that I've declared dependencies between tests implies that the success/failure of a test will be dictated by the success/failure of tests which it depends upon. That is certainly the case for me, where I use testng to drive three consecutive days of test file loads into a database. Day 3 depends on day 2, & day 2 depends upon day 1.

It would be great to hear from one of the core developers to find out whether the interleaving of test methods declared in different classes is truly expected/correct behaviour, or whether it is something that is going to be changed, and if it is going to be changed, when that will happen.

Thanks,

Andrew.

[juherr]

It would be great to hear from one of the core developers to find out whether the interleaving of test methods declared in different classes is truly expected/correct behaviour, or whether it is something that is going to be changed, and if it is going to be changed, when that will happen.

I'm already working on it, but I've no idea when something will be available because fixing the "issue" (which is not one) will open anothers: everything is about order and currently there is no priority rules between dependsOn, priority, preservOrder or groupByInstance.

[andrewah]

Thank you for letting me know that it is being worked on. I do appreciate that any change must be preceded by an impact analysis, that it can take time to resolve issues & that it is difficult to predict when resolutions to issues will be released. It is reassuring to know that you are investigating, irrespective of when a resolution will be made available.

However, with all due respect, the interleaving of test methods declared in different classes is an issue for me, my team & the project that I'm leading. Based on issue #106 evidently others consider the interleaving of test methods to be an issue as well.

Thanks,

Andrew.