XML attribute group-by-instances reduces number of tests run

[donovani]

TestNG Version

Encountered in v6.9.10

Expected behavior

Running XML test file containing group-by-instances="true" executes all listed tests, but reordered based on the individual instances.

Actual behavior

Tests with group-by-instances set to true will be reordered as expected, but some tests do not get executed. In the framework this was discovered, isolating the exact reason is challenging. The only discovery thus far is that the issue is resolved upon removing group-by-instances. The impact there is substantial. 3 classes never even get run, and those that do result in some test methods missing. I'll be looking into it further.

Is the issue reproductible on runner?

Yes. Experienced on the Eclipse plugin and while running through Maven builds.

Test case sample

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="testng" verbose="2">
    <!-- Bugged test. Remove group-by-instances and compare number of tests. -->
    <test name="gbi-bug" group-by-instances="true">
        <classes>
            <class name="tng.bug.StandaloneTestA" />
            <class name="tng.bug.StandaloneTestB" />
            <class name="tng.bug.TestFactory" />
        </classes>
    </test>
</suite>

StandaloneTestA.java

package tng.bug;

import org.testng.annotations.Test;

import static org.testng.Assert.assertTrue;

public class StandaloneTestA{

    @Test
    public void test1(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test2(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test3(){
        assertTrue( 0 < 1 );
    }
}

StandaloneTestB.java

package tng.bug;

import static org.testng.Assert.assertTrue;

import org.testng.annotations.Test;

public class StandaloneTestB{

    @Test
    public void test1(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test2(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test3(){
        assertTrue( 0 < 1 );
    }
}

TestFactory.java

package tng.bug;

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;

public class TestFactory {

    @Factory(dataProvider="testProvider")
    public Object[] testFactory(boolean positive) {
        if (positive) {
            return new Object[]{ new PositiveTest() };
        }
        else {
            return new Object[]{ new NegativeTest() };
        }
    }

    @DataProvider
    public Object[][] testProvider() {
        Object[][] data = new Object[30][1];
        for (int i=0; i<data.length; i++)
            data[i][0] = (i%2==0);
        return data;
    }
}

PositiveTest.java

package tng.bug;

import static org.testng.Assert.*;

import org.testng.annotations.Test;

public class PositiveTest {

    @Test
    public void test1(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test2(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test3() {
        assertTrue( 0 < 1 );
    }
}

NegativeTest.java

package tng.bug;

import static org.testng.Assert.assertTrue;

import org.testng.annotations.Test;

public class NegativeTest {

    @Test
    public void test1(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test2(){
        assertTrue( 0 < 1 );
    }

    @Test
    public void test3() {
        assertTrue( 0 < 1 );
    }
}

[donovani]

@juherr Thanks for taking the time on this. I reduced my code down today and added it to the initial post. This is the simplest I could bring it down to while still encountering the error. All of the test classes are similar - each having 3 simple tests to help track how many tests run/which got omitted.

The TestFactory also produces 30 tests (as written in the DataProvider method) because the bug seems to disappear at values less than that. I think that it may be some kind of memory limitation, or a problem when creating a graph of the tests at startup. If this doesn't work on your machine, I'd recommend cranking up the number of tests TestFactory creates to a significantly larger number before stating it doesn't work.

Thanks again! I appreciate the speed you and Cedric respond to issues with. 😃

[apetkova]

I can confirm I can still reproduce the issue with TestNG 6.11

[juherr]

@apetkova Ok, thanks. We will try to have a look on it.

@krmahadevan If you have some free time :)