Issue with test execution report when multiple scenario outlines and surefirererun is set

dunja132015 commented 1 month ago

👓 What did you see?

Have a feature test file like:

@XYZ
Feature: XYZ

  Background:
    Background steps

Scenario Outline: Failing test
    When step 1
    Then step a: "<a>"
    Examples:
      | a    |
      | string11 |
      | string12 |

Scenario Outline: Passing test
    When step 1
    Then step b: "<b>"
    Examples:
      | b     |
      | string21|
      | string22|

Java step Implementation:

@When("step1")
public void step1() {
    System.out.println("First step");
}

@Then("step a: {string}")
public void stepA(String a) {
    System.out.println(a);
    throw new NoSuchElementException();
}

@Then("step b: {string}")
public void stepB(String b) {
    System.out.println(b);
}

Execute the test from command line (make sure you have surefire.rerunFailingTestsCount set):

>mvn test -Dsurefire.rerunFailingTestsCount=2 -Dcucumber.filter.tags="@XYZ"

TEST OUTPUT:

[WARNING] ForkStarter IOException: Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty
Element name cannot be empty. See the dump file C:\path\target\surefire-reports\2024-08-07T16-54-16_637-jvmRun1.dumpstream
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

IMPORTANT: If the pipeline is used, test result is PASS (assuming due to 0 failures reported), and the next pipeline step is not blocked!!

NOTE: The issue seems to be already reported (and appearing fixed) here: https://github.com/cucumber/cucumber-jvm/issues/2709, but I'm reproducing it with the steps above. RunCucumberTest.xml file does not have the proper xml format, it ends with

<testcase name="Example #1.1" classname="Examples" time="0.408"

✅ What did you expect to see?

2 failed tests, 2 passed

📦 Which tool/library version are you using?

cucumber.version: 7.18.1 maven-surefire-plugin version: 3.3.1

🔬 How could we reproduce it?

No response

📚 Any additional context?

No response

mpkorstanje commented 1 month ago

I'm having some trouble reproducing your problem with the information provided.

Do you have a more precise minimal reproducer? You could fork the cucumber-java-skeleton and use that as a base.
Could you post the contents of the dumpstream file that was created?

dunja132015 commented 1 month ago

I updated the body:

Execute the test from command line (make sure you have surefire.rerunFailingTestsCount set):

>mvn test -Dsurefire.rerunFailingTestsCount=2 -Dcucumber.filter.tags="@XYZ"

Let me know if you can reproduce it with this information.

dunja132015 commented 1 month ago

2024-08-07T16-54-16_637-jvmRun1.dumpstream.txt

mpkorstanje commented 1 month ago

Unfortunately not. Could you make a minimal reproducer of the cucumber-java-skeleton?

mpkorstanje commented 1 month ago

The line numbers in the stack trace in the dump stream don't match the sources of maven-surefire-plugin version:3.3.1 so you may have another problem.

francislainy commented 2 weeks ago

Hi @mpkorstanje, I just would like to mention I'm also facing this same issue. My bom version is 7.18.0</cucumber.version> and I am using

<dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-spring</artifactId>
            <scope>test</scope>
</dependency>

<dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <scope>test</scope>
</dependency>

for <maven-surefire-plugin.version>3.1.0</maven-surefire-plugin.version> and <maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>. I'm trying as much as I can to try to find a reproducible issue for this but so far what I've noticed is that it seems to only happen for errors, not assertion failures.

This is my whole dump file:

# Created at 2024-08-28T16:25:32.681
ForkStarter IOException: Element name cannot be empty
Element name cannot be empty.
org.apache.maven.plugin.surefire.booterclient.output.MultipleFailureException: Element name cannot be empty
Element name cannot be empty
    at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.<init>(ThreadedStreamConsumer.java:59)
    at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer.<init>(ThreadedStreamConsumer.java:107)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:546)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
    Suppressed: java.lang.IllegalArgumentException: Element name cannot be empty
        at org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:430)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:294)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:204)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:158)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:51)
        at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59)
        at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86)
        at java.base/java.lang.Thread.run(Thread.java:833)
    Suppressed: java.lang.IllegalArgumentException: Element name cannot be empty
        at org.apache.maven.surefire.shared.utils.xml.PrettyPrintXMLWriter.startElement(PrettyPrintXMLWriter.java:245)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.getTestProblems(StatelessXmlReporter.java:430)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClassWithRerun(StatelessXmlReporter.java:294)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.serializeTestClass(StatelessXmlReporter.java:204)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:158)
        at org.apache.maven.plugin.surefire.report.StatelessXmlReporter.testSetCompleted(StatelessXmlReporter.java:51)
        at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSetCompleted(TestSetRunListener.java:193)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:143)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$TestSetCompletedListener.handle(ForkClient.java:127)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkedProcessEventNotifier.notifyEvent(ForkedProcessEventNotifier.java:197)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:303)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.handleEvent(ForkClient.java:59)
        at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:86)
        at java.base/java.lang.Thread.run(Thread.java:833)

Thank you.

The line numbers in the stack trace in the dump stream don't match the sources of maven-surefire-plugin version:3.3.1 so you may have another problem.

francislainy commented 2 weeks ago

PS: Just to add that this is how the xml file ends. (open test case tag and nothing else after it, this not even being the last test executed and which should be printed)

  <testcase name="Validate fields for total pans by risk against materialised views" classname="Validate the AMLAR top and total pans aggregation endpoints and their DB materialised views" time="2.795"/>
  <testcase name="Validate fields for total pans by risk against materialised views" classname="Validate the AMLAR top and total pans aggregation endpoints and their DB materialised views" time="2.795"/>
  <testcase name="Validate fields for total pans by risk against materialised views for dmp" classname="Validate the AMLAR top and total pans aggregation endpoints and their DB materialised views" time="0">
    <skipped message="&apos;cucumber.filter.tags=( ( @NotImporter and @hadoop ) and not ( @skip ) )&apos; did not match this scenario"/>
  </testcase>
  <testcase name="Example #1.1" classname="Examples" time="1.269"

Giving 0 as the number of retries does not seem to cause the issue, but anything other than that then it happens, unless all the tests pass on first attempt and a retry is not needed, so maybe something the retries are regenerating and corrupting the TEST.xml file?

mpkorstanje commented 2 weeks ago

@francislainy cheers!

I still can't reproduce the problem but I think I have a vague idea where the problem might be. One contributing factor seems to be that sure fire assumes test names to be unique. Can you try to reproduce your problem with all combinations of the following:

Use maven-surefire-plugin:3.5.0
Use maven-surefire-plugin:3.4.0
Ensure all your scenario and feature names are unique.

Set the cucumber.junit-platform.naming-strategy to long i.e:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.5.0</version>
<configuration>
    <properties>
        <configurationParameters>
            cucumber.junit-platform.naming-strategy=long
        </configurationParameters>
    </properties>
</configuration>
</plugin>

mpkorstanje commented 2 weeks ago

So the trick is to have a skipped test included with flaky test.

Reproducer:

Feature: Example Belly

  Scenario: a few cukes
    Given I have 0 cukes in my belly

  Scenario: a few cukes
    Given I have 21 cukes in my belly

  Scenario: a few cukes
    Given I have 42 cukes in my belly

package io.cucumber.skeleton;

import io.cucumber.java.en.Given;
import org.junit.jupiter.api.Assumptions;

public class StepDefinitions {
    @Given("I have {int} cukes in my belly")
    public void I_have_cukes_in_my_belly(int cukes) {
        if (cukes == 0) {
            return;
        }
        if (cukes == 21) {
            Assumptions.abort("Not now");
        }
        throw new RuntimeException("Oops");
    }
}

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <rerunFailingTestsCount>2</rerunFailingTestsCount>
                    <properties>

                        <!-- Work around. Surefire does not include enough
                             information to disambiguate between different
                             examples and scenarios. -->
                        <configurationParameters>
                            cucumber.junit-platform.naming-strategy=long
                        </configurationParameters>
                    </properties>
                </configuration>
            </plugin>

mpkorstanje commented 2 weeks ago

@francislainy @dunja132015 as a work around ensure that each feature has a unique name and that with in a feature all scenarios have unique names. Then also set cucumber.junit-platform.naming-strategy to long to ensure the examples in scenario out lines are given unique names.

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <version>3.5.0</version>
   <configuration>
       <properties>
           <configurationParameters>
               cucumber.junit-platform.naming-strategy=long
           </configurationParameters>
       </properties>
   </configuration>
</plugin>

mpkorstanje commented 2 weeks ago

@dunja132015 in addition, you'll also want to use JUnit 5 tag expressions to select your tests with Surefire instead of -Dcucumber.filter.tags="@XYZ". So you would use-DexcludedGroups="Ignore" -Dgroups="Smoke | Sanity".

See https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#tags.

francislainy commented 2 weeks ago

@francislainy cheers!

I still can't reproduce the problem but I think I have a vague idea where the problem might be. One contributing factor seems to be that sure fire assumes test names to be unique. Can you try to reproduce your problem with all combinations of the following:

Use maven-surefire-plugin:3.5.0

Use maven-surefire-plugin:3.4.0

Ensure all your scenario and feature names are unique.

Set the cucumber.junit-platform.naming-strategy to long i.e:
<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <version>3.5.0</version>
   <configuration>
       <properties>
           <configurationParameters>
               cucumber.junit-platform.naming-strategy=long
           </configurationParameters>
       </properties>
   </configuration>
</plugin>

@mpkorstanje Thanks a lot! Yes, allowing for the longer names seems to be a good workaround. However, the names get printed a bit toooooo long due to the example scenarios, but I guess we can live with this until a proper fix for the issue is found. But while on it, what I've also noticed looking at some more of our report failures is they seem to only happened for scenario outline and example pieces, that being the case when the testcase does not close and the report fails to print.

mpkorstanje commented 1 week ago

You can also make your scenarios unique with the pickle name strategy.

<configurationParameters>
 cucumber.junit-platform.naming-strategy=short
  cucumber.junit-platform.naming-strategy.short.example-name=pickle
 </configurationParameters>

But then you have to ensure that each of your scenario outlines uses a parameterized scenario name that is unique.

Scenario Outline: Failing test <a>
    When step 1
    Then step a: "<a>"
    Examples:
      | a    |
      | string11 |
      | string12 |

Scenario Outline: Passing test <b>
    When step 1
    Then step b: "<b>"
    Examples:
      | b     |
      | string21|
      | string22|

mpkorstanje commented 1 week ago

Created issue for Surefire: https://issues.apache.org/jira/browse/SUREFIRE-2260

But we'll probably have to add some suffixes to scenario names if they're not unique in Cucumber.

francislainy commented 3 days ago

You can also make your scenarios unique with the pickle name strategy.

<configurationParameters>
 cucumber.junit-platform.naming-strategy=short
  cucumber.junit-platform.naming-strategy.short.example-name=pickle
 </configurationParameters>

But then you have to ensure that each of your scenario outlines uses a parameterized scenario name that is unique.

Scenario Outline: Failing test <a>
    When step 1
    Then step a: "<a>"
    Examples:
      | a    |
      | string11 |
      | string12 |

Scenario Outline: Passing test <b>
    When step 1
    Then step b: "<b>"
    Examples:
      | b     |
      | string21|
      | string22|

Thank you @mpkorstanje . I'm not sure that would feasible in our case though, since a few of the scenarios require the same parameter name, such as . I've also looked into the opened issue for the surefire plugin and also not exactly sure this is only to do with duplicated test names since I've checked our feature file and there's no duplicates there. I'll try to spend some more time on it this week to see if I can spot anything that can help us with this issue. Other than this, it seems the number of tests run seems incorrect when there's a retry and it does not log the full amount of tests, but this would also require some further investigation and not sure whether we'd want to treat this as a separate issue.

francislainy commented 3 days ago

About the logging for the number of tests executed:

@Importer
Feature: Debug feature file

  @debug
  Scenario: A failed test
    And I force a failure to happen

  @debug
  Scenario: A passing test
    And I pass

public class ImporterStep {
    private static int runCount = 0;

    @Given("I pass")
    public void pass() {
        assertTrue(true);
    }

    @Given("I force a failure to happen")
    public void testFlaky() {
        runCount++;
        if (runCount == 1) {
            fail("Failing on the first attempt");
        }
        assertEquals(2, runCount);
        //        assertEquals(2, 1);
    }

Then I get this for the TEST.xml file, which shows 1 test run (as there's only 1 test which needs a rerunning), even though I have two tests in my feature file.

<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="com.mycompany.AmlarTest" time="0.238" tests="1" errors="0" skipped="0" failures="0">

mpkorstanje commented 3 days ago

Scenario outlines are syntactic sugar for repeating the same scenario several times with some variables replaced. Then because the pickle naming strategy is used, each example is named after the interpolated scenario outline (the thing you get when unrolling the scenario outline into individual scenarios). Because these are all the same, Surefire gets confused which scenario is actually being rerun.

jenisys commented 3 days ago

One other possibility to distinguish the different Scenarios generated by a Scenario Outline/Scenario Template is to use a FileLocation (like: some.feature:<LINE_NUMER>) that refers to the ExampleTable row that contains the parameters to generate this Scenario.

But I am not sure if that would have any effect with SUREFIRE.

The approach that @mpkorstanje suggested is probably better, like:

{ScenarioOutline.name}__@{ExamplesTable.index}_{ExampleTableRow.index} or
{ScenarioOutline.name}__@{ExampleTableRow.line_number} or
similar

(hint: multiple ExampleTable(s) may exist in a ScenarioOutline).

cucumber / cucumber-jvm