Incorrect thread creation using data provider with factory

StYura commented 2 years ago

TestNG Version 7.0 and 7.5

If Factory used with dataprovider with dataproviderthreadcount set to 5. Factory creates thread irrelevant to dataproviderthreadcount. Xml setup based on parallel="instances", "methods", "classes", "tests" are not helpful here cause though it changes behaviour it does not allow for multithreading based on dataprovider and sequential execution on test class level.

Expected behavior

Dataproviderthreadcount set to 5. Factory (to which data is provided) creates test classes based on processed dataprovider data. Tests getting executed in order of dataprovider provides data. In other words, expected that dataprovider is creating a thread with a single data set. That set of data is executed sequentially (see example) @DataProvider(parallel = true) public Object[][] getDummies2() { <- I`m expecting that here thread is created

@Factory(dataProvider = "getDummies2") <- and form here test is executed in single thread, sequentially for every element of List<String> params

Actual behavior

Dataprovider thread count set to 5. Factory (to which data is provided) creates test classes based on processed dataprovider data. All created tests are executed in the same thread pull regardless of dataprovider thread. Dataprovider does not seem to creat threads based on data it provides. Tests are getting executed based only on test classes created by Factory

Thread: Name: TestNG-test=Dummy test-1 Thread ID: 22 dummy = Three
key: key1 time = 00:48:01.797680600

Thread: Name: TestNG-test=Dummy test-3 Thread ID: 24 dummy = Four
key: key2 time = 00:48:01.797680600

Thread: Name: TestNG-test=Dummy test-2 Thread ID: 23 dummy = One
key: key1 time = 00:48:01.797680600

Thread: Name: TestNG-test=Dummy test-1 Thread ID: 22 dummy = Eleven
key: key4 time = 00:48:02.017808400

Thread: Name: TestNG-test=Dummy test-3 Thread ID: 24 dummy = Nine
key: key3 time = 00:48:02.017808400

Thread: Name: TestNG-test=Dummy test-2 Thread ID: 23 dummy = Seven
key: key3 time = 00:48:02.018742400

Thread: Name: TestNG-test=Dummy test-3 Thread ID: 24 dummy = Two
key: key1 time = 00:48:02.027744400

Thread: Name: TestNG-test=Dummy test-2 Thread ID: 23 dummy = Ten
key: key4 time = 00:48:02.032744600

Thread: Name: TestNG-test=Dummy test-1 Thread ID: 22 dummy = Eight
key: key3 time = 00:48:02.033863500

Thread: Name: TestNG-test=Dummy test-3 Thread ID: 24 dummy = Five
key: key2 time = 00:48:02.040741200

Thread: Name: TestNG-test=Dummy test-1 Thread ID: 22 dummy = Six
key: key2 time = 00:48:02.048749200

Thread: Name: TestNG-test=Dummy test-3 Thread ID: 24 dummy = Twelve
key: key4 time = 00:48:02.053744400

here we can see that tests with same key are getting executed at very same time e.g.

Thread: Name: TestNG-test=Dummy test-1 Thread ID: 22 dummy = Three
key: key1 time = 00:48:01.797680600

and

Thread: Name: TestNG-test=Dummy test-2 Thread ID: 23 dummy = One
key: key1 time = 00:48:01.797680600

different threads but same key

Is the issue reproducible on runner?

[ ] Shell
[x] Maven
[ ] Gradle
[ ] Ant
[ ] Eclipse
[x] IntelliJ
[ ] NetBeans

Test case sample

public class CompoundParallelDummyTest {
    @TestInstanceParameter
    private String param;
    private String key;

    public CompoundParallelDummyTest(String key, String input) {
        this.param = input;
        this.key = key;
    }

    @Test(description = "Create tests", threadPoolSize = 1, singleThreaded = true)
    public void createTests(){
        testDummy(param);
    }

    @Step("Execute orders for single user")
    public void testDummy(String param) {
        System.out.printf("Thread: Name: %s Thread ID: %d dummy = %s%nkey: %s time = %s\n\n",
                        Thread.currentThread().getName(),
                        Thread.currentThread().getId(),
                        param,
                        key,
                        LocalTime.now());
    }
}

public class DummyFactoryTest {

    @Factory(dataProvider = "getDummies2")
    public static Object[] createTests(String key,  List<String> params) {
        return params.stream().map(param -> new CompoundParallelDummyTest(key, param)).toArray();
    }

    @DataProvider(parallel = true)
    public Object[][] getDummies2() {
        List<String> params = List.of("One", "Two", "Three");
        List<String> params1 = List.of("Four", "Five", "Six");
        List<String> params2 = List.of("Seven", "Eight", "Nine");
        List<String> params3 = List.of("Ten", "Eleven", "Twelve");
        Map<String, Object> map = new HashMap<>();
        map.put("key1", params);
        map.put("key2", params1);
        map.put("key3", params2);
        map.put("key4", params3);

        return map.entrySet().stream()
                .map(e -> new Object[]{e.getKey(), e.getValue()})
                .toArray(Object[][]::new);
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="Dummy suite" data-provider-thread-count="3" preserve-order="true" thread-count="3" group-by-instances="true" parallel="instances">
    <test name="Dummy test" >
        <classes>
            <class name="dummy.DummyFactoryTest"></class>
        </classes>
    </test>
</suite>

Thanks!

bj-9527 commented 2 years ago

hi, could please specify where @Step("Execute orders for single user") comes from, seems not a supported annotation from TestNG.

StYura commented 2 years ago

hi, could please specify where @Step("Execute orders for single user") comes from, seems not a supported annotation from TestNG.

Yeah. @Step("Execute orders for single user") and @TestInstanceParameter are Allure report framework annotations. You can ignore(or delete) them completely.

krmahadevan commented 2 years ago

@StYura - Can you please help post a cleaned up test case, that can be used to simulate the problem ? One that doesn't use any other libraries etc., ?

Dataprovider thread count set to 5. Factory (to which data is provided) creates test classes based on processed dataprovider data. All created tests are executed in the same thread pull regardless of dataprovider thread. Dataprovider does not seem to creat threads based on data it provides. Tests are getting executed based only on test classes created by Factory

The attribute dataproviderthreadcount is intended to just ensure that the data provider starts providing test data in parallel instead of in a sequential order. This is usually relevant when you tie a dataprovider to a @Test method. This perhaps does not have any relevance when you tie it to a factory.

When you are working with factories that are powered by data provider, then you should be using parallel="instances" which will ensure that TestNG starts running all the tests in each of the instances in parallel.

Also if you are saying all this based on some documentation within TestNG, please help point me to it, because this looks to be a documentation bug.

In a nutshell.

attribute dataproviderthreadcount - Is relevant when you tie a @Test method with a data provider, so that the test method can be run for all the iterations from the data provider in parallel, instead of invoking the same test method for n sets of data, all one after the other.
attribute parallel=instances should be used when you have a factory that is powered by a dataprovider because here the data provider is going to be called in conjunction with a constructor, to create multiple instances and the parallel instances is now going to cause it to happen in parallel.

testng-team / testng