The @Test methods do not wait for @BeforeGroup to finish before they start in multi-threaded run

TestNG Version

Latest version: 7.5

Expected behavior

When u have multiple tests with @Test(group="group") methods and one @BeforeGroups(value = "group", alwaysRun = true) in a multi-threaded run with 4 threads, all of the @Test executions should start AFTER the @BeforeGroup method finishes its execution.

Actual behavior

Currently in multi-threaded execution with 4 threads the @BeforeGroups method starts, however in another thread also the execution of a @Test starts in the same time and does NOT wait for @BeforeGroups to finish before it starts.

Is the issue reproducible on runner?

[ ] Shell
[X] Maven - latest maven-surefire-plugin (3.0.0-M6)
[ ] Gradle
[ ] Ant
[ ] Eclipse
[X] IntelliJ
[ ] NetBeans

Test case sample

In a Maven project with latest TestNG(7.5) and latest maven-surefire-plugin(3.0.0-M6) have that code in a class named BeforeGroupsTest.java:

package tests.positive.folders;

import org.testng.annotations.BeforeGroups;
import org.testng.annotations.Test;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class BeforeGroupsTest {
    List<String> dataList;

    @BeforeGroups(value = "group", alwaysRun = true)
    public void setup() {
        System.out.println("setup");
        try {
            Thread.sleep(5000); //without sleep this bug doesnt reproduce
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        dataList = new ArrayList<>(Arrays.asList("A", "B")); //just an example
        System.out.println("setUp done");
    }

    @Test(groups = "group")
    public void testOne() {
        System.out.println("first test");
        String first = getStringFromList();
        System.out.println("first " + first);
    }

    @Test(groups = "group")
    public void testTwo() {
        System.out.println("second test");
        String second = getStringFromList();
        System.out.println("second " + second);
    }

    private String getStringFromList() {
        return dataList.stream().findFirst().orElse("C"); //just an example
    }
}

in your Terminal run the following Maven command:

mvn clean test -Dgroups="group" -Dthreads=4 you will observe a result like:

setup
first test
setUp done
second test
second A

[ERROR] tests.positive.folders.BeforeGroupsTest.testOne  Time elapsed: 0.002 s  <<< FAILURE!
java.lang.NullPointerException
        at tests.positive.folders.BeforeGroupsTest.getStringFromList(BeforeGroupsTest.java:40)
        at tests.positive.folders.BeforeGroupsTest.testOne(BeforeGroupsTest.java:28)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:135)
        at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:673)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:220)
        at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
        at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:945)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:193)
        at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
        at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

which clearly shows that the the "first test" did not wait for the @BeforeGroups to finish its execution before starting.

P.S. Have found that reporting under - https://github.com/cbeust/testng/issues/1859 , however I'm filing it as a new bug as I have just experienced it and looks like no fix was applied after that issues was filed in the past.

testng-team / testng