Getting unusual error while running parallel tests with data provider

[prasanta-biswas]

6.9.10

Expected behavior

All test cases should always pass.

Actual behavior

Sometimes all test cases pass and sometimes some of them fails due to java.lang.IndexOutOfBoundsException

Is the issue reproductible on runner?

• [ ] Shell
• [x] Maven
• [ ] Gradle
• [ ] Ant
• [ ] Eclipse
• [x] IntelliJ
• [ ] NetBeans

Test case sample

import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.text.NumberFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;

/**
 * Created by prasantabiswas on 23/02/17.
 */
public class Class1 extends TestBase
{
    private int a=20;
    @BeforeMethod(alwaysRun = true)
    public void beforeMethod(Object [] param)
    {
        if(param.length==0)
            param[0]="No Data";
        System.out.println("Class 1 Before Method Current Thread: "+Thread.currentThread().getId()+" Params: "+param[0]);
    }
    @AfterMethod(alwaysRun = true)
    public void afterMethod(ITestResult result)
    {
        String value;
        if(result.getParameters().length==0)
            value="No Para";
        else
            value=String.valueOf(result.getParameters()[0]);
        System.out.println("Class 1 After Method Current Thread: "+Thread.currentThread().getId()+" Params: "+value);
    }
    @Test(groups = {"group1"},dataProvider = "data1",dataProviderClass = DataProviders.class)
    public void test1(int i) throws ParseException
    {
        System.out.println("Class 1 test1 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
    @Test(groups = {"group2","group3"},dataProvider = "data2",dataProviderClass = DataProviders.class)
    public void test2(int i)
    {
        System.out.println("Class 1 test2 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
}

import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

/**
 * Created by prasantabiswas on 23/02/17.
 */
public class Class2 extends TestBase
{
    @BeforeMethod(alwaysRun = true)
    public void beforeMethod(Object[] param)
    {
        if(param.length==0)
            param[0]="No Data";
        System.out.println("Class 2 Before Method Current Thread: "+Thread.currentThread().getId()+" Params: "+param[0]);
    }
    @AfterMethod(alwaysRun = true)
    public void afterMethod(ITestResult result)
    {
        String value;
        if(result.getParameters().length==0)
            value="No Para";
        else
            value=String.valueOf(result.getParameters()[0]);
        System.out.println("Class 2 After Method Current Thread: "+Thread.currentThread().getId()+" Params: "+value);
    }
    @Test(groups = {"group3"},dataProvider = "data3",dataProviderClass = DataProviders.class)
    public void test3(int i)
    {
        System.out.println("Class 2 test3 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
    @Test(groups = {"group3"},dataProvider = "data4",dataProviderClass = DataProviders.class)
    public void test4(int i)
    {
        System.out.println("Class 2 test4 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
}

import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

/**
 * Created by prasantabiswas on 23/02/17.
 */
public class Class3 extends TestBase
{
    @BeforeMethod(alwaysRun = true)
    public void beforeMethod(Object [] param)
    {
        if(param.length==0)
            param[0]="No Data";
        System.out.println("Class 3 Before Method Current Thread: "+Thread.currentThread().getId()+" Params: "+param[0]);
    }
    @AfterMethod(alwaysRun = true)
    public void afterMethod(ITestResult result)
    {
        String value;
        if(result.getParameters().length==0)
            value="No Para";
        else
            value=String.valueOf(result.getParameters()[0]);
        System.out.println("Class 3 After Method Current Thread: "+Thread.currentThread().getId()+" Params: "+value);
    }
    @Test(groups = {"group4"},dataProvider = "data5",dataProviderClass = DataProviders.class)
    public void test5(int i)
    {
        System.out.println("Class 3 test5 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
    @Test(groups = {"group4"},dataProvider = "data6",dataProviderClass = DataProviders.class)
    public void test6(int i)
    {
        System.out.println("Class 3 test6 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
    @Test(groups = {"group5"},dataProvider = "data7",dataProviderClass = DataProviders.class)
    public void test7(int i)
    {
        System.out.println("Class 3 test7 Current Thread: "+Thread.currentThread().getId()+" i: "+i);
    }
}

import org.testng.annotations.DataProvider;
/**
 * Created by prasantabiswas on 29/03/17.
 */
public class DataProviders
{
    @DataProvider(name="data1",parallel=true)
    public static Object[][] data1()
    {
        return new Object[][]{{1},{2},{3},{4},{5}};

    }
    @DataProvider(name="data2",parallel=true)
    public static Object[][] data2()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
    @DataProvider(name="data3",parallel=true)
    public static Object[][] data3()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
    @DataProvider(name="data4",parallel=true)
    public static Object[][] data4()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
    @DataProvider(name="data5",parallel=true)
    public static Object[][] data5()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
    @DataProvider(name="data6",parallel=true)
    public static Object[][] data6()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
    @DataProvider(name="data7",parallel=true)
    public static Object[][] data7()
    {
        return new Object[][]{{1},{2},{3},{4}};

    }
}

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;

/**
 * Created by prasantabiswas on 23/02/17.
 */
public class TestBase
{
    private int a=10;
    @BeforeSuite(alwaysRun = true)
    public void beforeSuite()
    {
        System.out.println("Before Suite Method Called");
    }
    @AfterSuite(alwaysRun = true)
    public void afterSuite()
    {
        System.out.println("After Suite Method Called");
    }
}

I am running the following testng suite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="group-test" verbose="2" preserve-order="true"  thread-count="5" parallel="methods">
    <test name="testng-groups" preserve-order="true">
        <groups>
            <run>
                <include name="group1"/>
                <include name="group2"/>
                <include name="group3"/>
                <include name="group4"/>
            </run>
        </groups>
        <classes>
            <class name="Class1"/>
            <class name="Class2"/>
            <class name="Class3"/>
        </classes>
    </test>
</suite>

Sometimes i do not get any error and all test cases pass and sometimes some of the test cases fail. Here is the full output:

...
... TestNG 6.9.10 by Cédric Beust (cedric@beust.com)
...

[TestNG] Running:
  /Users/prasantabiswas/git/TestNGGroups/src/test/testng.xml
Before Suite Method Called
Class 1 Before Method Current Thread: 13 Params: 1
Class 1 Before Method Current Thread: 17 Params: 4
Class 1 Before Method Current Thread: 16 Params: 3
Class 1 Before Method Current Thread: 14 Params: 1
Class 1 Before Method Current Thread: 19 Params: 2
Class 1 Before Method Current Thread: 15 Params: 2
Class 1 Before Method Current Thread: 18 Params: 5
Class 1 Before Method Current Thread: 20 Params: 3
Class 1 test2 Current Thread: 13 i: 1
Class 1 Before Method Current Thread: 22 Params: 4
Class 2 Before Method Current Thread: 24 Params: 3
Class 2 Before Method Current Thread: 25 Params: 4
Class 2 Before Method Current Thread: 21 Params: 1Class 2 Before Method Current Thread: 23 Params: 2
Class 1 After Method Current Thread: 13 Params: 1
Class 1 test1 Current Thread: 14 i: 1
Class 1 test1 Current Thread: 16 i: 3
Class 1 After Method Current Thread: 14 Params: 1
Class 2 test3 Current Thread: 23 i: 2
Class 1 test1 Current Thread: 15 i: 2
Class 1 test1 Current Thread: 18 i: 5
Class 1 test2 Current Thread: 20 i: 3
Class 1 test2 Current Thread: 22 i: 4
Class 1 After Method Current Thread: 18 Params: 5
Class 2 test3 Current Thread: 24 i: 3
Class 2 test3 Current Thread: 25 i: 4
Class 2 test3 Current Thread: 21 i: 1
Class 2 After Method Current Thread: 23 Params: 2
Class 1 After Method Current Thread: 16 Params: 3
Class 2 After Method Current Thread: 25 Params: 4
Class 1 test1 Current Thread: 17 i: 4
Class 2 After Method Current Thread: 21 Params: 1
Class 1 After Method Current Thread: 22 Params: 4Class 2 After Method Current Thread: 24 Params: 3
Class 1 After Method Current Thread: 15 Params: 2
Class 1 After Method Current Thread: 20 Params: 3
Class 1 test2 Current Thread: 19 i: 2
Class 1 After Method Current Thread: 17 Params: 4
Class 1 After Method Current Thread: 19 Params: 2
Class 2 Before Method Current Thread: 29 Params: 2
Class 2 Before Method Current Thread: 27 Params: 1
Class 2 test4 Current Thread: 29 i: 2
Class 2 Before Method Current Thread: 30 Params: 3
Class 2 test4 Current Thread: 27 i: 1
Class 3 Before Method Current Thread: 31 Params: 1
Class 3 Before Method Current Thread: 34 Params: 3Class 2 After Method Current Thread: 29 Params: 2
Class 3 Before Method Current Thread: 35 Params: 4
Class 3 test5 Current Thread: 35 i: 4
Class 3 After Method Current Thread: 35 Params: 4
Class 3 Before Method Current Thread: 33 Params: 2
Class 3 test5 Current Thread: 34 i: 3

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

    at java.util.ArrayList.rangeCheck(ArrayList.java:653)
    at java.util.ArrayList.remove(ArrayList.java:492)
    at org.testng.IDEATestNGRemoteListener.onSuiteStart(IDEATestNGRemoteListener.java:167)
    at org.testng.IDEATestNGRemoteListener.onTestStart(IDEATestNGRemoteListener.java:197)
    at org.testng.IDEATestNGRemoteListener.onConfigurationStart(IDEATestNGRemoteListener.java:140)
    at org.testng.IDEATestNGInvokedMethodListener.beforeInvocation(IDEATestNGInvokedMethodListener.java:27)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithoutContextStrategy.callMethod(InvokedMethodListenerInvoker.java:84)
    at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:62)
    at org.testng.internal.Invoker.runInvokedMethodListeners(Invoker.java:556)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:493)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:215)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:589)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
    at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:75)
    at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Class 3 After Method Current Thread: 34 Params: 3
Class 3 test5 Current Thread: 31 i: 1
Test ignored.Empty test suite.

Test ignored.Class 2 After Method Current Thread: 27 Params: 1
Class 3 test5 Current Thread: 33 i: 2
Class 3 After Method Current Thread: 31 Params: 1
Class 2 After Method Current Thread: 30 Params: 3
Class 2 After Method Current Thread: 32 Params: 4
Class 3 After Method Current Thread: 33 Params: 2
Class 3 Before Method Current Thread: 36 Params: 1
Class 3 Before Method Current Thread: 38 Params: 3
Class 3 Before Method Current Thread: 39 Params: 4
Class 3 test6 Current Thread: 38 i: 3
Class 3 Before Method Current Thread: 37 Params: 2
Class 3 test6 Current Thread: 36 i: 1
Class 3 test6 Current Thread: 39 i: 4
Class 3 test6 Current Thread: 37 i: 2
Class 3 After Method Current Thread: 39 Params: 4
Class 3 After Method Current Thread: 37 Params: 2
Class 3 After Method Current Thread: 36 Params: 1
Class 3 After Method Current Thread: 38 Params: 3
[TestRunner] Starting executor for test testng-groups with time out:2147483647 milliseconds.After Suite Method Called

===============================================
group-test
Total tests run: 25, Failures: 0, Skips: 2
Configuration Failures: 1, Skips: 0
===============================================

Process finished with exit code 0

I don't understand why this is happening. I have attached the full project. Run it in IntelliJ with TestNG plugin. Any suggestion? TestNGGroups.zip

[krmahadevan]

@prasanta-biswas

From the stack trace you shared, it looks like a problem with the IntelliJ TestNG plugin (that piece is not owned by the TestNG project)

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

    at java.util.ArrayList.rangeCheck(ArrayList.java:653)
    at java.util.ArrayList.remove(ArrayList.java:492)
    at org.testng.IDEATestNGRemoteListener.onSuiteStart(IDEATestNGRemoteListener.java:167)
    at org.testng.IDEATestNGRemoteListener.onTestStart(IDEATestNGRemoteListener.java:197)
    at org.testng.IDEATestNGRemoteListener.onConfigurationStart(IDEATestNGRemoteListener.java:140)

Just to confirm this, can you please let us know if you are seeing this problem surface when you execute via one of the following means ?

• Via eclipse using the Eclipse TestNG plugin
• Via a build tool such as Gradle or Maven

If you are not seeing this problem through these means, but encountering this problem when only run via IntelliJ, then you may have to take this up with IntelliJ by logging a bug here

Looking at the codebase here it looks like there's perhaps a race condition of some sort.

[prasanta-biswas]

@krmahadevan I tried you suggestion and it did work there. It is problem of IntelliJ TestNG plugin. I will report a bug for this.

[krmahadevan]

@prasanta-biswas - Can we close off this issue in that case ?

[juherr]

@prasanta-biswas For reference, could you provide the link to the JetBrains issue?

[prasanta-biswas]

@juherr here