SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.22k stars 751 forks source link

Wrong test run data is passed to the Livingdoc report. #2746

Open smmaya opened 2 months ago

smmaya commented 2 months ago

SpecFlow Version

3.9.57

Which test runner are you using?

xUnit

Test Runner Version Number

2.5.8

.NET Implementation

.NET 6.0

Project Format of the SpecFlow project

Sdk-style project format

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

There seems to be a issue with what is being passed to the html report. Each time I run all tests (or a test, depending on hooks), a new TestExecution.html is created based on the TestExecution.json, nothing new here. BUT, what I see in the actual report is not the last run, but the one before the last one.

I have no idea if this is a date/time issue as the json "GenerationTime": "0001-01-01T00:00:00" has this value??? The time on the report is 2 minutes late as the files were created at 15h14 2024-04-25 15_15_08-net8 0

2024-04-25 15_27_33-TestExecutionReport html and 6 more pages - Work - Microsoft​ Edge

This is what I call it with: 2024-04-25 15_34_52-USC Schedule Integration Staging SpecFlowTest csproj - Microsoft Visual Studio

Any help is much appreciated.

Steps to Reproduce

Say you set the livingdoc hooks to [AfterFeature] to be faster so you don't wait for all tests to run And that we have those two tests:

Test A

Test B

Run Test A => if this is the first time you run it, no html report created yet, you will get an empty report Run Test B => now in the report you will see the test run results for the Test A 2024-04-25 15_16_32-_C__git_SpiralSuite_ScheduleStaging_Automation_USC Schedule Integration Staging

Link to Repro Project

No response

smmaya commented 1 month ago

OK, 2 weeks passed with no response, but I managed to find the cause for my livingdoc html report being created BEFORE the TestExecution.json being updated. In my LivingDocHooks class, I simply changed from process.WaitForExit(); to process.WaitForExitAsync(); and now I am getting the right data in the report. You can close this bug, thanks.