A passed test from a group is not included into testng-failed.xml for skipped test which is depended on this group

akrigator commented 1 year ago

TestNG Version

7.8.0

Note: only the latest version is supported

Expected behavior

The 7.3.0 version produces testng-failed.xml with passed, failed and skipped tests.

      <class name="com.test.DbgTest">
        <methods>
          <include name="test1"/>
          <include name="test2"/>
          <include name="test3"/>
        </methods>
      </class> <!-- com.test.DbgTest -->

Actual behavior

The 7.6.1 and 7.8.0 produces testng-failed.xml with failed and skipped tests.

      <class name="com.test.DbgTest">
        <methods>
          <include name="test3"/>
          <include name="test2"/>
        </methods>
      </class> <!-- com.test.DbgTest -->

Is the issue reproducible on runner?

[ ] Shell
[x] Maven
[ ] Gradle
[ ] Ant
[ ] Eclipse
[x] IntelliJ
[ ] NetBeans

Test case sample

package com.test;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class DbgTest {
    private String test1result;
    private String test2result;

    @Test(groups = "group")
    public void test1() {
        test1result = "something";
        assert true;
    }

    @Test(groups = "group")
    public void test2() {
        test2result = "something also";
        assert false;
    }

    @DataProvider
    public Object[][] dpResults() {
        return new Object[][] {
                { test1result },
                { test2result }
        };
    }

    @Test(dependsOnGroups = "group", dataProvider = "dpResults")
    public void test3(String result) {
        assert result != null;
    }
}

Suite xml

<?xml version="1.0" encoding="UTF-8"?>
<suite name="DBG Tests Suite" verbose="0" parallel="tests" thread-count="5" data-provider-thread-count="5">
    <test name="1" parallel="methods" thread-count="5">
        <classes>
            <class name="com.test.DbgTest"/>
        </classes>
    </test>
</suite>

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

juherr commented 1 year ago

It if not obvious but Groups are designed for test selection and order. There are not designed for test dependency.

In your sample, there is no reason to retry the succeed test.

If you use dependsOnMethods instead, it should work as you expect.

akrigator commented 1 year ago

@juherr, Here is an example from the 5.7.1 section of https://testng.org/doc/documentation-main.html:

@Test(groups = { "init" })
public void serverStartedOk() {}

@Test(groups = { "init" })
public void initEnvironment() {}

@Test(dependsOnGroups = { "init.*" })
public void method1() {}

Should a fail report include serverStartedOk or initEnvironment if some of them are failed? According to your comment - should not, but this example shows extremely clear that serverStartedOk and initEnvironment must be passed before running method1. No matter is it a first run or rerun of failed tests. More over the behaviour has been changed somewhere between 7.3.0 - 7.6.1 releases. Because of that i'm still guessing that this an defect.

juherr commented 1 year ago

It will depend on what groups are enabled in the runner.

Maybe in the sample, testng-failed.xml should include all the groups which were enabled and were a dependency of the failing test.

@krmahadevan wdyt?

krmahadevan commented 1 year ago

@akrigator - How are you running these tests ? I would expect TestNG to honour the group selection only if it was enabled as part of execution by filtering on groups. I remember streamlining some issues around the group selection which perhaps can explain why you see what you see.

Can you please edit your question and also include the suite xml file/surefire configuration or any other relevant details which will tell us more on the execution mode ?

krmahadevan commented 1 year ago

@akrigator - This is the issue that I was mentioning https://github.com/testng-team/testng/issues/2664

akrigator commented 1 year ago

@krmahadevan, i've added a suite xml example. Fill free to request anything also you need from me and thanks for quick involvement

krmahadevan commented 1 year ago

@akrigator - I was guessing that this would be the case :) (You are running without choosing any groups). If you go back and look at the issue I linked earlier, that was what was being discussed in that issue as well.

I think we eventually ended up restoring the old behaviour in TestNG, but sounds like the issue comes back when there are failures ?

I need to first check if dependsOnGroups is even being honoured here, because there's no clear behaviour clarity when it comes to groups and running a suite that doesnt say which group to run.

There are two interpretations when no group is specified:

I am not looking at filtering by groups. So ignore the notion of groups and run everything (This is what I am perceiving the behaviour should be)
Since I didn't mention any groups, run everything but honour the notion of groups (and thus dependsOnGroups)

@juherr - WDYT ? The tests are being run without any groups. So should we just think that we would like to ignore dependsOnGroups because there was no group selection involved at all ?

akrigator commented 1 year ago

From my points of view: TestNG provides ability to set dependencies via dependsOnMethods and dependsOnGroups attributes. DoG helps to simplify setting relations between tests if a dependent test is related to multiple tests, in comparing with DoM. For example, the test class could be overwritten to :

public class DbgTest {
    private String test1result;
    private String test2result;

    @Test
    public void test1() {
        test1result = "something";
        assert true;
    }

    @Test
    public void test2() {
        test2result = "something also";
        assert false;
    }

    @DataProvider
    public Object[][] dpResults() {
        return new Object[][] {
                { test1result },
                { test2result }
        };
    }

    @Test(dependsOnMethods = {"test1", "test2"}, dataProvider = "dpResults")
    public void test3(String result) {
        assert result != null;
    }
}

in this case a rerun of fails will work as expected. But this pattern is hard to be sustained in case of the "group" growing or increasing number of test dependent on the "group".

testng-team / testng