search

serenity-bdd / serenity-core

Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
http://serenity-bdd.info
Other
720 stars 519 forks source link

Scenario Outline being ignored during parallel test #2579

Open KurumiXenar opened 3 years ago

KurumiXenar commented 3 years ago

Hi, I'm using intellij with maven on this personal test project. I am using serenity-cucumber-6 for the tests scenarios.

When I try to run the tests in parallel with a batch count of 2 and applying a tag filter, serenity tells me this: There is a mismatch between the number of scenarios included in this test run ({}) and the expected number of scenarios loaded ({}). This suggests that the scenario filtering is not working correctly or feature file(s) of an unexpected structure are being run"

I have tried using the mvn clean verify with flags method: mvn clean verify -Dmode="parallel" -Dserenity.batch.count=2 -Dcucumber.filter.tags=@1234

I know Scenario Outlines can't be split internally to run in parallel but the expected result I am looking for during a parallel test run is for each parallel thread to run the scenario outline feature as a whole.

So for example, if I have 2 or more features in separate files tagged in the following way:

@test1234 Scenario Outline: Today is or is not Friday Given today is "" When I ask whether it's Friday yet Then I should be told ""

Examples: | day | answer | | Friday | Yes | | Sunday | No | | anything else! | No |

@test1234 Feature: Ability to share data between steps Scenario: Data is defined in different step classes Given that is empty and set as "hello" And that data is now "hello" And that data should be "hello" in another step When set data as "hello2" Then that data is now "hello2"

@test6789 Scenario Outline: Today is or is not Sunday Given today is "" When I ask whether it's Friday yet Then I should be told ""

Examples: | day | answer | | Monday | No | | Sunday | Yes | | anything else! | No |

Only tests tagged as @1234 should be run successfully by the parallel thread. It should not look at Scenario Outline and ignore it causing the scenario mismatch to appear and fail the run.

Is there a proper way for Scenario Outline to be treated as a whole scenario feature when parallel running? Thanks for the help.

wakaleo commented 3 years ago

This feature is community-proposed and largely community-driven, so if you are finding it useful your best bet would be to dig into the code and see what it does for yourself.