search

serenity-bdd / serenity-cucumber5

Other
14 stars 14 forks source link

What could be the reason for "mismatch between the number of scenarios" warning? #22

Open Jazzyekim opened 3 years ago

Jazzyekim commented 3 years ago

Hello,

We execute hundreds of tests in 16 threads. Tests are fetched from Jira(Xray). If all the tests that about to be executed are linked to one story - everything is fine, all tests will be executed. When we run tests linked to two or more stories some of the tests might not be executed. It is always different tests and different number of tests. I noticed the following warning in serenity log:

There is a mismatch between the number of scenarios included in this test run (21) and the expected number of scenarios loaded (17). This suggests that the scenario filtering is not working correctly or feature file(s) of an unexpected structure are being run

What could be the cause of such warning? The feature files seems to be valid. I have checked source code and found that this warning is thrown when the number of filteredInScenarioCount and number of weightedCucumberScenarios is different. Could you please clarify what are these two entities represent?

Jazzyekim commented 3 years ago

Have just tried to execute tests linked to only one story (it put all the tests into one feature file) and compared the feature file with the run where in addition some other tests were executed linked to another story ( two feature files were executed). It turned out that the feature files from these to runs are identical. So, it is not the problem with the structure of the file, but with the way how serenity handles splitting the tests from multiple feature files to run them in parallel