Introduce support for virtual threads

[sormuras]

Java 21 ships with virtual threads: https://openjdk.org/jeps/444

The JUnit Platform should provide an implementation of the HierarchicalTestExecutorService using virtual threads as an opt-in feature.

Deliverables

• [ ] Implement and test VirtualThreadHierarchicalTestExecutorService in the org.junit.platform.engine module

[jbduncan]

As a reminder, what is the advantage of running tests on virtual threads? Performance?

[kwakeroni]

As a reminder, what is the advantage of running tests on virtual threads? Performance?

I'm not the original poster, but I imagine that would be the main advantage, indeed.

Personally, I've imagined using virtual threads as a more dynamic alternative to @ResourceLock. @ResourceLock requires resources to be declared statically. Sometimes the resource is only known at runtime (for example due to configuration parameters).

Instead of grouping tests up-front by static resource, tests could try to get a lock on the resource they need and wait until it is available without blocking other tests. With the current thread pooling this could deteriorate the test suite to sequential execution, with virtual threads that would not be an issue. This way, custom extensions could hook into the concurrent execution simply by 'holding up' a test until it is ready to be run.

ParallelExecutionConfiguration seems rather tied to the thread pooling concept. Having a configuration parameter to select a custom HierarchicalTestExecutorService implementation, would already allow for easier experimentation. (referring to the implementation of JupiterTestEngine.createExecutorService)

[goughy000]

Do we think something along these lines (but more polished up of course) would be a useful first step to allow selecting a custom HierearchicalTestExecutorService via configuration parameters, without getting into the weeds of virtual threads specifically?

As @kwakeroni says this would allow people to do some experimentation with virtual threads (or other things) without introducing them into jupiter itself yet

(I've not put much thought into this or really tried it out, just copied how it works for the TempDirFactory)

diff --git a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/JupiterTestEngine.java b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/JupiterTestEngine.java
index dbd799b7f5..cea6dc9d0b 100644
--- a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/JupiterTestEngine.java
+++ b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/JupiterTestEngine.java
@@ -22,6 +22,7 @@ import org.junit.jupiter.engine.descriptor.JupiterEngineDescriptor;
 import org.junit.jupiter.engine.discovery.DiscoverySelectorResolver;
 import org.junit.jupiter.engine.execution.JupiterEngineExecutionContext;
 import org.junit.jupiter.engine.support.JupiterThrowableCollectorFactory;
+import org.junit.platform.engine.ConfigurationParameters;
 import org.junit.platform.engine.EngineDiscoveryRequest;
 import org.junit.platform.engine.ExecutionRequest;
 import org.junit.platform.engine.TestDescriptor;
@@ -30,6 +31,7 @@ import org.junit.platform.engine.support.config.PrefixedConfigurationParameters;
 import org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService;
 import org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine;
 import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorService;
+import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorServiceFactory;
 import org.junit.platform.engine.support.hierarchical.ThrowableCollector;

 /**
@@ -73,11 +75,13 @@ public final class JupiterTestEngine extends HierarchicalTestEngine<JupiterEngin
        @Override
        protected HierarchicalTestExecutorService createExecutorService(ExecutionRequest request) {
                JupiterConfiguration configuration = getJupiterConfiguration(request);
-               if (configuration.isParallelExecutionEnabled()) {
-                       return new ForkJoinPoolHierarchicalTestExecutorService(new PrefixedConfigurationParameters(
-                               request.getConfigurationParameters(), Constants.PARALLEL_CONFIG_PREFIX));
+               if (!configuration.isParallelExecutionEnabled()) {
+                       return super.createExecutorService(request);
                }
-               return super.createExecutorService(request);
+
+               return configuration.getHierarchicalTestExecutorServiceFactory().get().createExecutorService(
+                       new PrefixedConfigurationParameters(request.getConfigurationParameters(),
+                               Constants.PARALLEL_CONFIG_PREFIX));
        }

        @Override
diff --git a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/CachingJupiterConfiguration.java b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/CachingJupiterConfiguration.java
index 2d61b58c1c..8d343b988f 100644
--- a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/CachingJupiterConfiguration.java
+++ b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/CachingJupiterConfiguration.java
@@ -29,6 +29,7 @@ import org.junit.jupiter.api.extension.ExecutionCondition;
 import org.junit.jupiter.api.io.CleanupMode;
 import org.junit.jupiter.api.io.TempDirFactory;
 import org.junit.jupiter.api.parallel.ExecutionMode;
+import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorServiceFactory;

 /**
  * Caching implementation of the {@link JupiterConfiguration} API.
@@ -125,4 +126,10 @@ public class CachingJupiterConfiguration implements JupiterConfiguration {
                        key -> delegate.getDefaultTempDirFactorySupplier());
        }

+       @SuppressWarnings("unchecked")
+       @Override
+       public Supplier<HierarchicalTestExecutorServiceFactory> getHierarchicalTestExecutorServiceFactory() {
+               return (Supplier<HierarchicalTestExecutorServiceFactory>) cache.computeIfAbsent(PARALLEL_EXECUTION_EXECUTOR_SERVICE_PROPERTY_NAME,
+                               key -> delegate.getHierarchicalTestExecutorServiceFactory());
+       }
 }
diff --git a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/DefaultJupiterConfiguration.java b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/DefaultJupiterConfiguration.java
index d64c4ceee3..e435171c1b 100644
--- a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/DefaultJupiterConfiguration.java
+++ b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/DefaultJupiterConfiguration.java
@@ -32,6 +32,7 @@ import org.junit.jupiter.api.parallel.ExecutionMode;
 import org.junit.platform.commons.util.ClassNamePatternFilterUtils;
 import org.junit.platform.commons.util.Preconditions;
 import org.junit.platform.engine.ConfigurationParameters;
+import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorServiceFactory;

 /**
  * Default implementation of the {@link JupiterConfiguration} API.
@@ -62,6 +63,10 @@ public class DefaultJupiterConfiguration implements JupiterConfiguration {
        private static final InstantiatingConfigurationParameterConverter<TempDirFactory> tempDirFactoryConverter = //
                new InstantiatingConfigurationParameterConverter<>(TempDirFactory.class, "temp dir factory");

+       private static final InstantiatingConfigurationParameterConverter<HierarchicalTestExecutorServiceFactory> testExecutorServiceFactoryConverter = //
+               new InstantiatingConfigurationParameterConverter<>(HierarchicalTestExecutorServiceFactory.class,
+                       "test executor service factory");
+
        private final ConfigurationParameters configurationParameters;

        public DefaultJupiterConfiguration(ConfigurationParameters configurationParameters) {
@@ -141,4 +146,11 @@ public class DefaultJupiterConfiguration implements JupiterConfiguration {
                return () -> supplier.get().orElse(TempDirFactory.Standard.INSTANCE);
        }

+       @Override
+       public Supplier<HierarchicalTestExecutorServiceFactory> getHierarchicalTestExecutorServiceFactory() {
+               Supplier<Optional<HierarchicalTestExecutorServiceFactory>> supplier = testExecutorServiceFactoryConverter.supply(
+                       configurationParameters, PARALLEL_EXECUTION_EXECUTOR_SERVICE_PROPERTY_NAME);
+
+               return () -> supplier.get().orElse(HierarchicalTestExecutorServiceFactory.Standard.INSTANCE);
+       }
 }
diff --git a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/JupiterConfiguration.java b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/JupiterConfiguration.java
index 559b4d7d57..9637a30c3b 100644
--- a/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/JupiterConfiguration.java
+++ b/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/config/JupiterConfiguration.java
@@ -27,6 +27,7 @@ import org.junit.jupiter.api.io.CleanupMode;
 import org.junit.jupiter.api.io.TempDirFactory;
 import org.junit.jupiter.api.parallel.Execution;
 import org.junit.jupiter.api.parallel.ExecutionMode;
+import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorServiceFactory;

 /**
  * @since 5.4
@@ -36,6 +37,7 @@ public interface JupiterConfiguration {

        String DEACTIVATE_CONDITIONS_PATTERN_PROPERTY_NAME = "junit.jupiter.conditions.deactivate";
        String PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME = "junit.jupiter.execution.parallel.enabled";
+       String PARALLEL_EXECUTION_EXECUTOR_SERVICE_PROPERTY_NAME = "junit.jupiter.execution.parallel.executorservice";
        String DEFAULT_EXECUTION_MODE_PROPERTY_NAME = Execution.DEFAULT_EXECUTION_MODE_PROPERTY_NAME;
        String DEFAULT_CLASSES_EXECUTION_MODE_PROPERTY_NAME = Execution.DEFAULT_CLASSES_EXECUTION_MODE_PROPERTY_NAME;
        String EXTENSIONS_AUTODETECTION_ENABLED_PROPERTY_NAME = "junit.jupiter.extensions.autodetection.enabled";
@@ -70,4 +72,6 @@ public interface JupiterConfiguration {

        Supplier<TempDirFactory> getDefaultTempDirFactorySupplier();

+       Supplier<HierarchicalTestExecutorServiceFactory> getHierarchicalTestExecutorServiceFactory();
+
 }

[kwakeroni]

Is there anything I can do to help this move along ?

[Chetan33]

This looks promising. Can we get a snapshot version to try on? Since it may take time before it gets merged.

[marcphilipp]

We discussed this issue in the team recently and came to the conclusion that we should at least have one demo/scenario where using virtual threads to execute tests actually improves performance before publishing this feature.

@Chetan33 Do you have something like that in mind?

[kwakeroni]

@marcphilipp I've mentioned a use case a while ago.

While it is not about virtual threads per se, it is about having more control over the parallelization of tests in a dynamic way. Virtual threads would be a good fit for my use case.

Personally, I've imagined using virtual threads as a more dynamic alternative to @ResourceLock. @ResourceLock requires resources to be declared statically. Sometimes the resource is only known at runtime (for example due to configuration parameters).

Instead of grouping tests up-front by static resource, tests could try to get a lock on the resource they need and wait until it is available without blocking other tests. With the current thread pooling this could deteriorate the test suite to sequential execution, with virtual threads that would not be an issue. This way, custom extensions could hook into the concurrent execution simply by 'holding up' a test until it is ready to be run.

ParallelExecutionConfiguration seems rather tied to the thread pooling concept. Having a configuration parameter to select a custom HierarchicalTestExecutorService implementation, would already allow for easier experimentation. (referring to the implementation of JupiterTestEngine.createExecutorService)

[marcphilipp]

That would require the JVM to be able to detect a virtual thread being blocked. The use cases for resource locks go beyond that since the "resource" might be external to the JVM, e.g. a file to which concurrent modification should be prevented.

On a related note, you might be interested in #2677 for which a PR is currently in review.

[kwakeroni]

That would require the JVM to be able to detect a virtual thread being blocked. The use cases for resource locks go beyond that since the "resource" might be external to the JVM, e.g. a file to which concurrent modification should be prevented.

Well detecting that is basically what virtual threads are about.

I think virtual threads are mostly helpful to simplify managing the access to the resources, whatever they are. If each resource is protected by something like a semaphore, the test framework can simply launch all the tests without further need to orchestrate the access to the resources.

I haven't tried this, so maybe I'm overlooking something. If I find some time, I could try the above patch on a local branch and see what it gives. (let me know if the team is interested)

On a related note, you might be interested in #2677 for which a PR is currently in review.

I'll definitely check that one out, thanks 👍

[nickh-stripe]

I was recently reaching for virtual threads support as an option to work around the sequential execution and impact of ResourceLock, atm if I have 500 tests with locks and 500 without, and 10 normal executing threads it is likely (excluding manual ordering etc) that my 10 executing threads will be blocked waiting for the resource lock instead of progressing the 500 lock-free tests instead. The naive way to workaround this is to increase the number of executing threads to be greater than the number of locks so some of the lock-free tests will always progress, but depending on the number of locks creating hundreds of executing threads to ensure this behaviour, is possible but expensive, when something like virtual threads exists to solve this exact problem.

[marcphilipp]

@nickh-stripe Did you restrict the maximum number of threads? We use ForkJoinPool.ManagedBlock under the hood so a blocked thread should be compensated by potentially spawning another.

[nickh-stripe]

yes in this scenario there is an upper limit on threads, our tests consume external resources so can't be unbounded.

I only comment as there was a call out upthread for a scenario where virtual threads helps "actually improves performance". What i'm trying to achieve is possible without virtual threads (upto a memory limit..) but it can be quite expensive, virtual threads makes it much cheaper which seems like an improvement.

It also seems to fit with the current level of control available in the framework, there is not a huge amount of control over how tests are actually executed, outside of static ahead of time configuration there aren't that many options to dynamically effect the way tests are executed which pushes users down the 'throw more threads at it' path like in my example, so if that is the preferred approach then making that approach cheaper seems logical. 👍

[marcphilipp]

Thanks for your input! We'll experiment with that scenario once we pick up this issue again.

[Chetan33]

We discussed this issue in the team recently and came to the conclusion that we should at least have one demo/scenario where using virtual threads to execute tests actually improves performance before publishing this feature.

@Chetan33 Do you have something like that in mind?

First of all I apologies for being so late to reply. Being swamped with so many responsibility both at work and home. We have a service which is pure functional and we are writing tests. Ideally each test can be run concurrently. We are already using threaded concurrency however, as I understand it is thread pool joined so it waits to start new requests, typically based on the number of cores. Even if the tweak Number of threads in config I don't see any improvement compared to default strategy . I was wondering if number of concurrent requests can be increased using virtual threads if not unlimited but in thousands.