SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.25k stars 754 forks source link

SpecFlow + LivingDoc -> LivingDoc shows the test failed, even though it passed after the retry. #2703

Open darthftw opened 1 year ago

darthftw commented 1 year ago

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

4.4.0.0

.NET Implementation

.NET 6.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

{ "NRetrySettings": { "maxRetries": 3, "applyGlobally": true } }

Issue Description

Hello,

I apologize in advance if this isn't considered a SpecFlow bug but rather a livingdoc one, but the issue I'm having is this.

I have this simple test which verifies if the user is authenticated, and if not, it will call the re-auth method and retrieve the token, and then it will retry the scenario again using NUnitRetry.SpecFlowPlugin - 3.9.59.

The test shows as passed in the log, but whenever I try to generate the livingdoc html report, it shows up as failed.

Output LivingDoc

I've tried the : https://docs.specflow.org/projects/specflow-runner/en/latest/Profile/VSTest.html but to no avail, using a Default.srprofile as well as the .runsettings one.

I do believe I'm doing something wrong, but I can't warp my head around what, maybe someone else encountered this, and has a workaround?

Steps to Reproduce

@Retry(n) for a scenario which fails the first time, but then succeeds the second time, and then generate the livingdoc report.

Link to Repro Project

No response

arpitsinghal3432 commented 8 months ago

I am facing the same issue. I see this bug is open for a long time, can we expect a solution for this anytime soon ?