New tests don't start until set of running tests are finished (serenity + junit5 + cucumber7)

[p-nikolaichik]

👓 What did you see?

New tests don't start until set of running tests are finished. I start 1000 scenarios in 50 threads in parallel by scenarios and I noticed that time of execution is 4 hour (for comparison if I run the same suite of tests using jvm parallel plugin in parallel by feature files then total time is 1 hour). I built graphs and saw that new tests are not started even if previous are finished. It means that the threads are just idle waiting for the rest of the tests instead of starting new ones. I created two the similar simple test projects for comparison. First with serenity, second with with poor selenium and web driver manager. Project with simple selenium doesn't have such issue. See following graphs.

✅ What did you expect to see?

If test is finished new test is started immediately without waiting for other tests finish

junit-platform.properties:

cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=fixed
cucumber.execution.parallel.config.fixed.parallelism=50
cucumber.execution.parallel.config.fixed.max-pool-size=50
cucumber.plugin=io.cucumber.core.plugin.SerenityReporterParallel
cucumber.filter.tags=not @Skip and not @Manual

   <serenity.version>3.6.12</serenity.version>
    <junit.platform.version>1.9.3</junit.platform.version>
    <cucumber.junit.platform.engine.version>7.11.2</cucumber.junit.platform.engine.version>
    <junit.jupiter.migration.support>5.0.0-M4</junit.jupiter.migration.support>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-cucumber</artifactId>
        <version>${serenity.version}</version>
    </dependency>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-junit5</artifactId>
        <version>${serenity.version}</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit-platform-engine</artifactId>
        <version>${cucumber.junit.platform.engine.version}</version>
    </dependency>
    <dependency>
        <groupId>org.junit.platform</groupId>
        <artifactId>junit-platform-suite</artifactId>
        <version>${junit.platform.version}</version>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-migration-support</artifactId>
        <version>${junit.jupiter.migration.support}</version>
    </dependency>
    <dependency>
        <groupId>net.serenity-bdd</groupId>
        <artifactId>serenity-rest-assured</artifactId>
        <version>${serenity.version}</version>
    </dependency>

Environment: Reproduced with Serenity 3.6.12/3.9.7, junit4/5, cucumber7, java 11/17 https://github.com/p-nikolaichik/serenity-cucumber-junit5

Serenity

Selenium with web driver manager

@wakaleo Maybe I missed some necessary config parameter to run tests in parallel?

[wakaleo]

Have a look at https://github.com/serenity-dojo/swaglabs-serenity-cucumber for an example of parallel execution with Cucumber and Serenity BDD.

[p-nikolaichik]

What I see on this graph:

1. The same gaps in each thread between tests like on my first screenshot.
2. Next set of tests started when almost all previous tests are finished. Tests in the next set are started at the same time and it can be seen that all tests are run like in batches and a new batch of tests is not launched until almost the entire previous batch of tests is completed.

I expected behavior like on my second screenshot: As soon as test is finished new test should be started immediately without any delay cos it leads to significant time increasing.

@wakaleo Was that how it was originally intended? Can I make the tests run like in my second screenshot?

[wakaleo]

I really don't know, have a look in the code but the parallel execution is not managed by Serenity, it all happens at the Cucumber/JUnit level. All Serenity does is record the results and aggregate them at the end.

[p-nikolaichik]

But we don't have such issue if we use selenium instead of serenity

[ordeh]

I faced with the same issue. @wakaleo could you clarify how it works?

According to this comment https://github.com/serenity-bdd/serenity-core/issues/3202#issuecomment-1659996048 we should not se differences between selenium and serenity. However it is not true and parallel execution with serenity performs worse(

[wakaleo]

@cliviu might have some thoughts, but feel free to investigate the code to see if you can propose any improvements. I have no idea how Cucumber handles the parallel execution, but the Serenity integration is in the SerenityReporterParallel class. There may be some syncing issue. @p-nikolaichik If you would like someone to investigate further, you are certainly welcome to get your company to obtain a commercial support contract so we can spend some time on this, or dig into the code yourself.

[wakaleo]

I faced with the same issue. @wakaleo could you clarify how it works?

According to this comment #3202 (comment) we should not se differences between selenium and serenity. However it is not true and parallel execution with serenity performs worse(

Maybe you can have a look at the SerenityReporterParallel class, or maybe run your tests with a profiler to see if you can see any locking issues.

[p-nikolaichik]

@wakaleo @cliviu The reason is clearly related to this class, because if I remove io.cucumber.core.plugin.SerenityReporterParallel from junit-platform.properties, then the result is excellent, there are no any issues with parallel execution. However report is not generated in this case.

Serenity without SerenityReporterParallel

[wakaleo]

SerenityReporterParallel is what provides the Serenity reporting and API support. What are you replacing SerenityReporterParallel with?

[wakaleo]

This may be non-trivial.@p-nikolaichik if this feature is of value for your company, there is always the option of requesting commercial support. This would allow us to dedicate time to research the issue in depth, in particular in the context of your environment. Would you be up for this?

[cliviu]

@p-nikolaichik , how do you get those nice charts?