Default behavior of dependsOnMethods is unexpected

[vguna]

TestNG Version

6.10

Expected behavior

When running ITs from the maven failsafe plugin with default settings (no testng.xml specified, no parallel or forking configured etc.), I would expect, that all tests are running in order. That means grouped by class and optionally respecting dependsOnMethods.

Actual behavior

Currently running ITs shows, that as soon as using dependsOnMethods, these methods are not run in order. Instead these are postponed somehow and get executed concurrently with other tests, which is not expected.

I started to explicitly use a testng.xml and use preserve-order etc. without luck. Then I stumbled across:

https://github.com/cbeust/testng/issues/261 and especially https://github.com/cbeust/testng/commit/4bcbae0733e52fc32182a8fd4cc70553cedc9e3d

After setting group-by-instance explicitly on suite, the problem is gone. This is my current replacement for the default failsafe invocation:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="all-tests" verbose="1" parallel="false" preserve-order="true" group-by-instances="true">
    <test name="all-tests">
        <packages>
            <package name="com.acme.*" />
        </packages>
    </test>
</suite>

Now, I would expect, if no special configuration is given, that all tests run in order - no matter what. Is the current implementation by design? Thanks for your support!

BTW: testng complains about parallel=false since none is the current way to go. Is there an updated dtd avalable?

Is the issue reproductible on runner?

• [ ] Shell
• [x] Maven
• [ ] Gradle
• [ ] Ant
• [ ] Eclipse
• [ ] IntelliJ
• [ ] NetBeans

Test case sample

See https://github.com/juherr/testng/commit/4bcbae0733e52fc32182a8fd4cc70553cedc9e3d

[vguna]

As it seems (https://github.com/cbeust/testng/issues/952) this is the current expected behavior. Well I would think for most of the DEVs not ;).

[cbeust]

You're not giving much details about what ordering you get versus what ordering you expected.

The semantics of dependsOnMethods is very clear: if method a depends on method b, then b must run before a. That's it. There is no other specification, it's the only requirement. If you have an example demonstrating that this behavior is not honored, please post it here.

Thanks!

[vguna]

My expectations would be, that all tests within a class are executed first, before executing tests of another class. Often tests setup a specific set of data that the test methods are using. E.g. we're setting up a GreenMail mail server in @BeforeClass and stop it in @AfterClass. After adding more tests/classes, we saw that there were binding issues during test execution. It took us some to time to figure out that this was only caused when @dependsOnMethods was used. Hence the unexpected behavior from our point of view during execution. Methods without it are running as expected. What I basically want to say is, that the definition of @dependsOnMethods might be correct, but the side effects of using it (all methods of a class are not executed together anymore) wasn't obvious to us. Maybe in the end it is just a lack of documentation, pointing out these effects :)?

[juherr]

@vguna Your expectations look legit but due to dependencies between methods the order may change. So, the order depends on the context and that's why it is unspecified. We can imagine having a default dependency between methods from the same class.