Parallelism value ignored for the fixed strategy

[rbok78]

Steps to reproduce

git clone git@github.com:rbok78/framework-template.git cd framework-template mvn clean test -P parallel-execution

Actual outcome

All 4 test methods are kicked off at the same time.

Expected outcome

I use the following configuration for the Maven Surefire Plugin. I expect the number of test methods running in parallel is the same as the parallelism value. I've tried to set it to 1, 2, and 3 but still all 4 test methods run in parallel.

junit.jupiter.execution.parallel.enabled = true junit.jupiter.execution.parallel.mode.default = concurrent junit.jupiter.execution.parallel.mode.classes.default = concurrent junit.jupiter.execution.parallel.config.strategy = fixed junit.jupiter.execution.parallel.config.fixed.parallelism = 1

Context

• Linux 5.3.0-46-generic x86_64
• Ubuntu 19.10
• OpenJDK 11.0.7
• Apache Maven 3.6.1
• Maven Surefire Plugin 3.0.0-M4
• Jupiter 5.7.0-M1
• IntelliJ IDEA 2020.1 (Community Edition)

Deliverables

n/a

[marcphilipp]

Thanks for reporting the issue and providing a reproducer! 👍

The parallelism setting is no guarantee that there will be at most one test executing simultaneously. It has the same semantics as ForkJoinPool.getParallelism(), i.e. a potentially blocking operation may lead to another thread being used in order to ensure progress. If you don't want your test methods to run in parallel, please use the following configuration:

junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent

[rbok78]

Thanks for the answer. I've done some reading about the ForkJoinPool and I understand that the number of threads created can be higher than the parallelism setting. Still I would expect that that number of running tests (concurrently running threads) would follow the parallelism setting. I plan to use JUnit5 for my UI automation where I can have hundreds of tests (test methods that can be within a single class or multiple classes). I can't run all at the same time as it would exhaust all hardware resources hence I need to be able to control how many tests run concurrently. I stripped my code to the bare minimum and discovered a weird behaviour. For the parallelism setting:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = 2

Without creating an instance of the ChromeDriver anywhere in my code, the number of test methods that run concurrently follows the parallelism setting:

11:58:19.135 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for pageTitleTest
11:58:19.135 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for failTestWithScreenshot
11:58:19.164 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for failTestWithScreenshot
11:58:19.203 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for pageTitleTest
11:58:19.305 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for waitForRequestTest
11:58:19.305 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for matchTemplateTest
11:58:19.319 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for matchTemplateTest
11:58:19.322 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for waitForRequestTest

However, when I create an instance of the ChromeDriver e.g. in the method annotated with BeforeEach or in the test method directly, I can see all test methods are kicked off at the same time:

12:07:13.622 [ForkJoinPool-1-worker-13] INFO  org.example.utils.BaseTest - Set up completed for getDomHighResTimeStampTest
12:07:13.619 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Set up completed for pageTitleTest
12:07:13.620 [ForkJoinPool-1-worker-7] INFO  org.example.utils.BaseTest - Set up completed for linkTextTest
12:07:13.615 [ForkJoinPool-1-worker-5] INFO  org.example.utils.BaseTest - Set up completed for matchTemplateTest
12:07:13.621 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Set up completed for failTestWithScreenshot
12:07:13.693 [ForkJoinPool-1-worker-15] INFO  org.example.utils.BaseTest - Set up completed for waitForNetworkIdleTest
12:07:13.698 [ForkJoinPool-1-worker-9] INFO  org.example.utils.BaseTest - Set up completed for paragraphTextTest
12:07:13.704 [ForkJoinPool-1-worker-11] INFO  org.example.utils.BaseTest - Set up completed for waitForRequestTest
12:07:13.707 [ForkJoinPool-1-worker-3] INFO  org.example.utils.BaseTest - Tear down completed for failTestWithScreenshot
12:07:13.736 [ForkJoinPool-1-worker-15] INFO  org.example.utils.BaseTest - Tear down completed for waitForNetworkIdleTest
12:07:13.736 [ForkJoinPool-1-worker-13] INFO  org.example.utils.BaseTest - Tear down completed for getDomHighResTimeStampTest
12:07:13.794 [ForkJoinPool-1-worker-11] INFO  org.example.utils.BaseTest - Tear down completed for waitForRequestTest
12:07:13.818 [ForkJoinPool-1-worker-7] INFO  org.example.utils.BaseTest - Tear down completed for linkTextTest
12:07:13.823 [ForkJoinPool-1-worker-5] INFO  org.example.utils.BaseTest - Tear down completed for matchTemplateTest
12:07:13.828 [ForkJoinPool-1-worker-1] INFO  org.example.utils.BaseTest - Tear down completed for pageTitleTest
12:07:13.830 [ForkJoinPool-1-worker-9] INFO  org.example.utils.BaseTest - Tear down completed for paragraphTextTest

The only way to fix this, which I have found so far, is to use a custom strategy. With this one in place, I can control the number of running tests reliably. Here is the parallelism setting:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = custom
junit.jupiter.execution.parallel.config.custom.class = org.example.CustomStrategy

And here is the custom strategy with the parallelism set to 2:

package org.example;

import org.junit.platform.engine.ConfigurationParameters;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfiguration;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfigurationStrategy;

public class CustomStrategy implements ParallelExecutionConfiguration, ParallelExecutionConfigurationStrategy {
    @Override
    public int getParallelism() {
        return 2;
    }

    @Override
    public int getMinimumRunnable() {
        return 2;
    }

    @Override
    public int getMaxPoolSize() {
        return 2;
    }

    @Override
    public int getCorePoolSize() {
        return 2;
    }

    @Override
    public int getKeepAliveSeconds() {
        return 30;
    }

    @Override
    public ParallelExecutionConfiguration createConfiguration(final ConfigurationParameters configurationParameters) {
        return this;
    }
}

I will try to reach out to the ChromeDriver developers if they can help to explain the current behaviour.

[marcphilipp]

Thanks for the detailed analysis and posting the workaround!

My only hunch is that the ChromeDriver instantiation somehow behaves differently when invoked from a ForkJoinWorkerThread. If ForkJoinPool.managedBlock is used somehow behind the scenes the ForkJoinPool will preventively start another thread to prevent blocking.

[kobynet]

I can confirm we are having the same issue, not using any ChromeDriver at all. using OpenJDK 11.0.9, Jupiter 5.7.0 and Maven Surefire Plugin 3.0.0-M5

I can confirm the workaround showed in https://github.com/junit-team/junit5/issues/2273#issuecomment-623850954 works for us as well.

Our confguration:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
junit.jupiter.execution.parallel.config.strategy = fixed
junit.jupiter.execution.parallel.config.fixed.parallelism = 2

What we see is all classes are triggered at once and not just 2 at a time.

[titusfortner]

@rbok78 / @kobynet were you by chance using Selenium 4 when you had this issue?

[shs96c]

If I understand the above comments correctly, does this mean that if any code under test (either first or via dependency) calls ForkJoinPool.managedBlock the parallelism count used by junit is basically ignored? If that's the case, then I'd suggest we reopen this issue.

Put another way, something in Selenium's deps may have tickled this issue, but it's something that could affect any JUnit 5 user.

[shs96c]

Having a hunt in the JDK, this should impact anyone who's using Process.onExit(), or CompleteableFuture.get(), the latter of which is used pretty widely.

[kobynet]

@titusfortner We're note using any selenium deps.

[titusfortner]

Of course, you even said that in the first line of your comment; sorry. :)

I guess I should also say that the proposed solution only seems to work if I want to limit the number of parallel executions to a very small number. When increasing it to a reasonable number, it does not work.

Here's a sample project showing the problem, with the relevant config values: https://github.com/titusfortner/bug9359/blob/main/pom.xml#L62-L79

I really want to be able to recommend JUnit 5 to my clients because I love the new syntax/functionality, but I can't until this gets sorted.

[marcphilipp]

I'm all ears if anyone has a proposal how to resolve this. The only thing I've come up with would be to wrap the execution of each test in another, non-ForkJoinPool thread but that would double the number of required threads and have performance implications.

[titusfortner]

Out of curiosity, how does JUnit 4 / maven surefire approach it? That has always worked extremely well for our purposes.

[titusfortner]

@marcphilipp would it work to just use something like AtomicInteger to limit the total number of running threads?

[marcphilipp]

You can already limit the number of threads by implementing a custom ParallelExecutionConfigurationStrategy that returns a ParallelExecutionConfiguration where getMaxPoolSize() returns the limit. However, since a ForkJoinPool is used in the background a small number might lead to blockages in combination with @ResourceLock and other synchronization mechanisms.

[titusfortner]

well, damn it all. I was using a custom strategy and it wasn't working in Selenium beta 3, but guess what? It works in Selenium beta 4.

Thanks for your reply, @marcphilipp !

[sangameshwar-balur]

@rbok78 / @kobynet were you by chance using Selenium 4 when you had this issue?

I am having same issue when I am using Selenium 4, here what could be the issue with Selenium 4. Any suggestion to resolve the issue with Selenium 4.

[titusfortner]

So, it turns out that it works with Java 11, but not Java 8 or Java 17. Discussion from Selenium here:  https://github.com/SeleniumHQ/selenium/issues/10113 Which references open JUnit 5 issues: #2787 & #2792

[harrietwamala]

In my case junit parameters did not work, what worked was only the cucumber parameters below: cucumber.execution.parallel.enabled = true cucumber.execution.parallel.mode.default = same_thread cucumber.execution.parallel.mode.classes.default = concurrent cucumber.execution.parallel.config.strategy = fixed cucumber.execution.parallel.config.fixed.parallelism = 2

[titusfortner]

fwiw, I got this working for today on Java 8 with JUnit 4.9.2 I'm not sure how/why it works on Java 8, but 🤷‍♂️ Method level parallelization, not just class level.

<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.config.strategy = fixed
    junit.jupiter.execution.parallel.config.fixed.parallelism = 10
    junit.jupiter.execution.parallel.config.fixed.max-pool-size = 10
</configurationParameters>

[BalurQA]

@titusfortner : 4.9.2 is the version of Junit 5 or Junit4 . ? I do not see this version 4.9.2 in Maven repository for either. Could you please help with maven gradle dependency details from maven repo.

[titusfortner]

Sorry, 5.9.2

[BalurQA]

@titusfortner I did try with 5.9.2 unfortunately it is not working for me. If possible could you please share list of dependencies used in the project or working git repo.

Below are list dependencies used in my project framework. Kindly let me know of any concerns with dependencies used

dependencies {
//    testImplementation 'org.junit.jupiter:junit-jupiter:5.9.2'
//    testImplementation('org.junit.jupiter:junit-jupiter:5.5.1')
    testImplementation(platform('org.junit:junit-bom:5.9.2'))
    testImplementation('org.junit.jupiter:junit-jupiter')
// https://mvnrepository.com/artifact/io.github.bonigarcia/selenium-jupiter
    implementation 'io.github.bonigarcia:selenium-jupiter:4.3.1'
    testImplementation 'org.hamcrest:hamcrest:2.1'
    testImplementation 'org.hamcrest:hamcrest-library:2.1'
//    testCompile 'org.junit.jupiter:junit-jupiter-api:5.9.2'
//  testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.9.2'
    testRuntime 'org.junit.platform:junit-platform-launcher:1.4.2'
    testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.9.2'
    compile group: 'org.jsoup', name: 'jsoup', version: '1.9.2'
    compile group: 'org.apache.poi', name: 'poi', version: '3.9'
    compile group: 'org.apache.poi', name: 'poi-contrib', version: '3.6'
//    compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.141.59'
    // https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java
    implementation 'org.seleniumhq.selenium:selenium-java:4.8.0'
    compile group: 'io.qameta.allure', name: 'allure-junit5', version: '2.11.0'
    compile group: 'org.apache.pdfbox', name: 'pdfbox', version: '2.0.16'
    compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.17.1'
    compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.17.1'
    compile group: 'io.qameta.allure', name: 'allure-gradle', version: '2.7.0'
    compile group: 'org.apache.poi', name: 'poi-ooxml', version: '3.7'
    compile group: 'org.apache.xmlbeans', name: 'xmlbeans', version: '4.0.0'
    compile group: 'com.automation-remarks', name: 'video-recorder-junit5', version: '2.+'
    compile('com.assertthat:selenium-shutterbug:0.9.2')
    compile 'org.slf4j:slf4j-nop:1.7.25'
//  compile 'com.microsoft.ews-java-api:ews-java-api:2.0'
    compile group: 'org.jsoup', name: 'jsoup', version: '1.9.2'
    compile group: 'com.surevine', name: 'ews-java-api', version: '2.3'
    compile group: 'ru.yandex.qatools.ashot', name: 'ashot', version: '1.5.4'
//    implementation group: 'io.github.bonigarcia', name: 'webdrivermanager', version: '4.3.1'
    implementation group: 'javax.mail', name: 'mail', version: '1.4.7'
    implementation group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
    runtimeClasspath group: 'javax.mail', name: 'javax.mail-api', version: '1.6.2'
    implementation group: 'org.json', name: 'json', version: '20201115'
    implementation group: 'com.googlecode.json-simple', name: 'json-simple', version: '1.1.1'
    implementation group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '7.2.2.jre8'
    implementation 'net.java.dev.jna:jna-platform:5.9.0'
    // https://mvnrepository.com/artifact/org.junit.platform/junit-platform-engine
    testImplementation 'org.junit.platform:junit-platform-engine:1.9.0'
    compile group: 'com.epam.healenium', name: 'healenium-web', version: '3.2.5'
    // https://mvnrepository.com/artifact/com.fasterxml.jackson/jackson-bom
    implementation 'com.fasterxml.jackson:jackson-bom:2.14.1'
    compile group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.4'
}

[Kolesnikov-Vladislav]

@BalurQA

У меня с 5.9.2 работает =)

Мои зависимости:

            "com.codeborne:selenide:6.12.4",
            "org.junit.jupiter:junit-jupiter:5.9.2",
            "org.aspectj:aspectjweaver:1.9.19",
            "org.slf4j:slf4j-simple:2.0.7",
            "org.aeonbits.owner:owner:1.0.12",
            "com.github.javafaker:javafaker:1.0.2",
            "io.qameta.allure:allure-selenide:2.21.0",
            "io.qameta.allure:allure-rest-assured:2.21.0",
            "org.testng:testng:7.1.0",
            "com.codeborne:selenide-proxy:6.12.4"

[paajake]

fwiw, I got this working for today on Java 8 with JUnit 4.9.2 I'm not sure how/why it works on Java 8, but 🤷‍♂️ Method level parallelization, not just class level.

<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.config.strategy = fixed
    junit.jupiter.execution.parallel.config.fixed.parallelism = 10
    junit.jupiter.execution.parallel.config.fixed.max-pool-size = 10
</configurationParameters>

This worked for me using JDK 21, with JUnit 5.10.2, the fixed.parallelism value still seems ignored, but the fixed.max-pool-size variable seems to keep things in check

[mbuchner]

I can confirm that this seams to be a Java 17 issue - upgraded now to Java 21 and used the following configuration to run always 2 classes in parallel:

    -Djunit.jupiter.execution.parallel.enabled=true 
    -Djunit.jupiter.execution.parallel.mode.default=same_thread 
    -Djunit.jupiter.execution.parallel.mode.classes.default=concurrent 
    -Djunit.jupiter.execution.parallel.config.fixed.max-pool-size=2 
    -Djunit.jupiter.execution.parallel.config.strategy=fixed 
    -Djunit.jupiter.execution.parallel.config.fixed.parallelism=2

Quarkus, jUnit 3.8.2, Selenide 7.2.3, Selenium Grid with 2 Chrome nodes (Concurrency 1)