Setting stopAtFirstError to true has no effect.

[Xen0byte]

SpecFlow Version

3.9.52

Which test runner are you using?

NUnit

Test Runner Version Number

3.13.2

.NET Implementation

.NET 6.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer + Command Line via dotnet test

SpecFlow Section in app.config or content of specflow.json

{ "language": { "feature": "en-GB", "tool": "en-GB" },

"bindingCulture": {
    "name": "en-GB"
},

"generator": {
    "allowDebugGeneratedFiles": true,
    "allowRowTests": true,
    "addNonParallelizableMarkerForTags": [
        "Solo"
    ]
},

"runtime": {
    "missingOrPendingStepsOutcome": "Pending",
    "obsoleteBehavior": "Warn",
    "stopAtFirstError": true
},

"trace": {
    "traceSuccessfulSteps": true,
    "traceTimings": true,
    "minTracedDuration": "0:0:0.01",
    "stepDefinitionSkeletonStyle": "RegexAttribute"
},

"stepAssemblies": [
    {
        "assembly": "DRAMA"
    },
    {
        "assembly": "REHEARSAL"
    }
],

"knowledgeBase": [
    {
        "specflowConfiguration": "https://docs.specflow.org/projects/specflow/en/latest/Installation/Configuration.html"
    }
]

}

Issue Description

After setting stopAtFirstError to true all the subsequent steps will still run. Additionally, it is not clear whether stopAtFirstError applies within the scope of the scenario context, the feature context, or the test run context.

Steps to Reproduce

1. Create a feature file along the lines of...

@DEBUG
Feature: Navigate To GitHub
    AS A PROGRAMMER WHO DILIGENTLY DEBUGS THEIR CODE
    I SHOULD BE ABLE TO NAVIGATE TO A WEBSITE PASSED AS A PARAMETER

    Scenario:   [01/02] Navigate To Website
        When    I NAVIGATE TO "https://github.com/"
        Then     I CONFIRM THAT THE PAGE TITLE CONTAINS "GitHub"

    Scenario:   [02/02] Take A Screenshot
        Then     I TAKE A SCREENSHOT

2. Cause the first step to fail with an error.
3. Run the tests and observe that steps 2 and 3 will still be executed.

Link to Repro Project

No response

[SabotageAndi]

stopAtFirstError is on the scenario level and not on a test run.

From the docs:

Determines whether the execution should stop when encountering the first error, or whether it should attempt to try and match subsequent steps (in order to detect missing steps).

I made it a little bit clearer in the documentation.

[Xen0byte]

Hi @SabotageAndi. Updating the documentation doesn't address the fact that the setting doesn't seem to work as intended. As described in the summary, steps after the first failed step are still being executed.

[SabotageAndi]

I can't reproduce this behavior you describe. The project I created is here: https://github.com/SabotageAndi/GH2563

[Xen0byte]

Aha, I'll clone that maybe later today or sometime tomorrow and see if I can repro it on my end.

[Xen0byte]

OK, I've had another look and it would appear that the feature is working fine in the scope of the scenario context. I suppose that previously it wasn't entirely clear what the scope was, which is why I initially though there might have been an issue here, but now it is, as per the updated documentation.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.