Batching Fails when Scenarios of same Feature are present in different files

[sudarsunperu]

Hi Team, If I put Scenarios belonging to the same Feature in separate files, then the batch execution fails. I tested this behaviour for serenity-core from 2.4.24 and 3.1.5 and its the same.

e.g.: Say there are two feature files Sceanrio1.feature and Scenario2.feature.

Scenario1.feature

Feature: Searching for a term

  Scenario: Searching for a term
    Given Sergey is researching things on the internet
    When he looks up "Cucumber"
    Then he should see information about "Cucumber"

Scenario2.feature

Feature: Searching for a term

  Scenario: Searching for a term again
    Given Sergey is researching things on the internet
    When he looks up "Serenity"
    Then he should see information about "Serenity"

when I execute it I get the error as stated in the below details.

gradlew clean test -Pdriver=firefox --info -Dserenity.batch.count=2 -Dserenity.batch.number=1

The error log

starter.CucumberTestSuite STANDARD_OUT
    14:29:44.334 [Test worker] INFO  i.c.junit.CucumberSerenityRunner - Running slice 2 of 2 using fork 1 of 1 from feature paths [file:/C:/Users/M1032630/Downloads/serenity-cucumber-starter-1.0.0/src/test/resources/features/]
    14:29:44.481 [Test worker] ERROR i.c.junit.CucumberSerenityRunner - Test failed to start
    java.lang.IllegalArgumentException: Can't find feature 'Searching for a term #2' in this slice
        at net.serenitybdd.cucumber.suiteslicing.WeightedCucumberScenarios.createFilterContainingScenariosIn(WeightedCucumberScenarios.java:69)
        at io.cucumber.junit.CucumberSerenityRunner.lambda$toPossibleFeatureRunner$5(CucumberSerenityRunner.java:328)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

Observe that #⁠2 gets appended to the feature name java.lang.IllegalArgumentException: Can't find feature 'Searching for a term #2' in this slice

However if I put both the scenarios in a single file Scenarios.feature the batching and execution works perfectly fine.

Feature: Searching for a term

  Scenario: Searching for a term
    Given Sergey is researching things on the internet
    When he looks up "Cucumber"
    Then he should see information about "Cucumber"

  Scenario: Searching for a term again
    Given Sergey is researching things on the internet
    When he looks up "Serenity"
    Then he should see information about "Serenity"

For maintainability and reporting purpose, my project maintains each Scenario in a separate .feature file and all Scenarios of the same feature are kept in a directory.

Please suggest to fix this batching issue when maintaining scenarios of same feature in a differnet files.

[wakaleo]

Try without batches.

[sudarsunperu]

Hi @wakaleo , yes without batches it works fine. But my project has 350 + test cases, and we use batching for parallel execution.

Currently it uses the below versions and Batching works fine when maintaining each Scenario in a separate file.

I'm trying to update the versions to the latest without breaking my pipeline for parallel execution using Batching.

 slf4jVersion = '1.7.7'
    serenityCoreVersion = '2.0.41'
    serenityCucumberVersion = '1.9.31'
    serenityRestAssuredVersion = '2.0.40'
    junitVersion='4.12'
    assertJVersion='3.8.0'
    logbackVersion='1.2.3'
    webDriverManagerVersion='4.1.0'

[wakaleo]

The batch handling is community-driven, so you may have to dig into the code to see what it is doing.

[wakaleo]

Why don't you just run the features in parallel using the Cucumber support for parallel execution?

[sudarsunperu]

Thanks @wakaleo , I will try to look into the Code and Provide a PR. Alternately, I will also look into parallel execution using Cucumber for gradle.

Because of OOM exceptions when running 350+ tests in the same machine, we used batching and executed each batch in a separate node. However I'm not sure if the same is possible using Cucumber parallel execution

[harrypreston-dlg]

I just wanted to add I have also encountered this exact issue and it also does the same sort of thing if the name of the features file are the same but in different folders within the cucumber feature source . For example ./folder1/purchases.feature and ./folder2/purchases.feature would cause both of them to be looked for in both batch slices and then it will fail because it can't find the name in the feature slice it was given (because its not actually there) .

[keshrey]

HI @sudarsunperu , Any progress on this , Did you able to run tests in batches with upgrading to newer version of serenity and cucumber ?

[wakaleo]

This isn't supported, feature names must be unique. Batches are probably also made redundant by Cucumber 7 parallel execution.