cucumber / cucumber-jvm

Cucumber for the JVM
https://cucumber.io
MIT License
2.69k stars 2.02k forks source link

ConcurrentEventListener handler for TestStepStarted event is accessing steps after scenario failure #2855

Open Venomicon opened 4 months ago

Venomicon commented 4 months ago

Hi, we're using ConcurrentEventListener to access Step text via TestStepStarted event listener. These steps can later be logged to a file and attached to the failed test in the Allure report. Unfortunately, it does not work as expected.

👓 What did you see?

All steps for given scenario are being logged, even the ones that occur after the scenario failure.

##[SECTION]  Starting scenario d950c44e-1029-40ae-8896-91c1c72414ec
##[STEP]     02/29/2024 09:53:11 --- action ---
##[STEP]     02/29/2024 09:53:11 --- check result ---
##[ERROR]    02/29/2024 09:53:11 --- Step above is failing! ---
##[STEP]     02/29/2024 09:53:11 --- rest ---
##[STEP]     02/29/2024 09:53:11 --- of ---
##[STEP]     02/29/2024 09:53:11 --- the ---
##[STEP]     02/29/2024 09:53:11 --- scenario ---
##[SECTION]  Scenario ended with status: FAILED

java.lang.AssertionError: 
Expecting value to be true but was false

    at org.steps.CommonSteps.checkResult(CommonSteps.java:163)
    at ✽.check result(file:///C:/Projects/ConcurrentEventListenerBug/src/test/java/org/features/ConcurrentEventListener.feature:440)

Failed scenarios:
file:///C:/Projects/ConcurrentEventListenerBug/src/test/java/org/features/ConcurrentEventListener.feature:438 # ConcurrentEventListener bug

✅ What did you expect to see?

I expect, that steps after the failed step are not logged, because as name suggests for "TestStepStarted" event - these steps have not yet started because of the failure.

📦 Which tool/library version are you using?

<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-java</artifactId>
    <version>7.15.0</version>
</dependency>
<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-testng</artifactId>
    <version>7.15.0</version>
</dependency>
<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-picocontainer</artifactId>
    <version>7.15.0</version>
</dependency>

🔬 How could we reproduce it?

Given scenario:

Scenario: ConcurrentEventListener bug
    When action
    Then check result <---- this step is failing
    Given rest
    When of
    And the
    Then scenario

ConcurrentEventListener implementation:

public class StepLogger implements ConcurrentEventListener {

    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestStepStarted.class, this::handleTestStepStarted);
    }

    private void handleTestStepStarted(TestStepStarted testStepStartedEvent) {
        if (testStepStartedEvent.getTestStep() instanceof PickleStepTestStep testStep) {
            Step currentStep = testStep.getStep();
            Log.step(currentStep.getText());
        }
    }
}

cucumber.properties:

cucumber.publish.quiet=true
cucumber.snippet-type=camelcase
cucumber.plugin=summary, html:target/reports/cucumber-pretty.html, json:target/reports/cucumber/cucumber.json, \
  io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm, org.utils.StepLogger
mpkorstanje commented 4 months ago

As part of the execution model, every step is always executed. But every step after a failing step is executed in dry-run mode and will not invoke the glue code. I.e. it is skipped.

https://github.com/cucumber/cucumber-jvm/blob/9fc324247408d751bd371f823bc842eae26201b6/cucumber-core/src/main/java/io/cucumber/core/runner/TestCase.java#L82-L86

The reasons for this are pretty much historic now. They have their origins in the original Ruby implementation. The test step event docs could probably be updated to include that information.

For now you may want to use the TestStepFinished event instead and filter on it's result status.