mpkorstanje
commented
4 days ago Very interesting. Unfortunately, I'm just as puzzled as you are. And the backlog of things to do for Cucumber well exceeds the time I can reasonably spend on it. So your active participation will be required for a resolution.
According to the results I obtained, I would like to describe a scenario to explain what I believe might be happening
Your theory seems consistent with observations. But a few points of note:
-
Your Cucumber and JUnit versions are not to up to date. Please ensure this problem is reproducible with the latest version. These are 7.18.0 and 5.10.3 respectively. Additionally, make sure your dependencies align by using the Cucumber and JUnit BOMs.
-
Cucumber uses JUnit which uses a
ForkJoinPoolto execute tests in parallel. This pool potentially interacts with other fork join pools. Potentially from other frameworks, the test code, or the system under test. I find https://github.com/junit-team/junit5/issues/3108#issuecomment-1520174615 and the surrounding comments to be quite insightful.At a glance, the behavior you are observing looks like it could be work stealing and this is under other circumstances the expected behavior. But I don't have plausible of explanation of how that might exactly be happening.
So a good first step would be to create a minimal reproducer based on the cucumber-java-skeleton. This should rule out any confounding factors. If your problem can not be reproduced this way, you will have the unenviable task of paring down your companies code until it is a minimal reproducer that can be shared. But a good reproducer is often also the quickest way to get something in JUnit 5 fixed, though again here too your active participation will likely be required.
You may also want to check your code for other interactions with ForkJoinPool. This could potentially rule out a category of problems.
I am not entirely familiar with the inner workings of the cucumber engine. I will try to reach a conclusion based on the basic concepts of concurrent execution that I am familiar with and the way I perceive the resources might be allocated. I hope it might help to speed up the resolution of the issue, since I spend some time trying to decipher what might be occurring and how I could tackle this.
Let me get you up to speed on the architecture.
The Cucumber JUnit Platform Engine uses the JUnit Platform. The JUnit Platform provides an API for describing and executing tests. Tests are represented by a tree of Node objects. The EXECUTION_MODE_FEATURE_PROPERTY_NAME property is translated to an ExecutionMode. This is set on each each Node below the node for the Feature file, When set to same_thread this node should be executed in the same thread as it's parent.
When executing the cucumber-junit-platform-engine Cucumber does not manage parallelism itself. Instead it uses a ForkJoinPoolHierarchicalTestExecutorService provided by JUnit.
So any debugging efforts you'll probably want to focus on
ForkJoinPoolHierarchicalTestExecutorService.submit and ForkJoinPoolHierarchicalTestExecutorService.invokeAll.
Andre-MR-Pereira-NBI
👓 What did you see?
Result
Small test suite
Output is: [Feature file name] || Scenario name => Thread Id
This is company code, and I am not exactly sure what I can share about it, so I took the liberty to alter the names of the results. However, this is what I see when I run a small test suite in parallel, with a fixed strategy, with 4 threads and the execution mode as same_thread. I tracked the results by creating a concurrent Queue, where I store the feature file, scenario name and thread id in the @ Before Hook. I highlighted and organized the results to be easier to read:
Large test suite
This behavior also appear in a larger test suite, namely, when testing 335 scenarios over 50 feature files. Again, the test has the exact same cucumber options has the small test suite, and for simplicity sake I am going to give a summary of the results:
✅ What did you expect to see?
I expect to see ALL feature files being ran by the parent thread when utilizing the cucumber.execution.execution-mode.feature=same_thread option.
📦 Which tool/library version are you using?
Language
Java 17
Compiler
Maven
Docker image
maven:3.8.5-openjdk-17-slim
Junit Version
5.9.2
Cucumber Version
7.3.0
🔬 How could we reproduce it?
Mock
Create a test suite split into several distinct feature files, with a couple scenarios in each of the feature files;
Create a junit-platform.properties file in the src/test/resources/ folder, and fill it with this information:
Run the mvn test command!
📚 Any additional context?
Pre discussion
Theory
According to the results I obtained, I would like to describe a scenario to explain what I believe might be happening:
Structure
Execution
In broader terms, it is generally great to have a pool of resources that we can assign work to as soon as a resource frees up, ensuring we maximize their usage and they spend the least amount of time idling, so we can finish our workload faster. However, looking at the task we have at hand, we really don't want at any point in time for two (or more) threads to utilize the same data within a feature file. If the data is not properly taken care before being used in a scenario, our tests will fail with erroneous results.
Possible Action
Is it possible to ensure that when there are no more feature files to be ran, the scenarios that are still waiting to be executed from other feature files will not grab a thread from the resources pool, and just wait for the parent thread of that feature file to execute them?