Double execution with xUnit using Test Explorer

mocsharp commented 2 years ago

SpecFlow Version

3.9.52

Which test runner are you using?

xUnit

Test Runner Version Number

3.9.52

.NET Implementation

.NET 5.0

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

None

Issue Description

When the test project is executed using Test Explorer, each and every test is executed twice.

Immediately after Test Explorer shows complete execution, in the background, the test is executed a 2nd time.

Steps to Reproduce

Using the VS template & the calculate example, update GivenTheFirstNumberIs(...) with the following code

            using var stream = File.CreateText($"C:\\some-directory\\test-{DateTime.Now.Ticks}.txt");
            stream.WriteLine(DateTime.Now.ToString());
            stream.Close();

And remove all the throw new PendingStepException lines. Click Run all test in view in Test Explorer

Expected Behavior:

one single file is created

Actual Behavior:

two files are created (2nd file is created within 3-5 seconds after 1st)

Link to Repro Project

No response

mocsharp commented 2 years ago

Update: If there are multiple test cases defined and you selected to run just one test case from the Test Explorer. Upon completing the test case, all of them are executed in the background.

SabotageAndi commented 2 years ago

Please try to create a project where we can reproduce this issue. There has to be something in the project what causing this. I am 99% sure the double execution of a test isn't SpecFlow fault.

mocsharp commented 2 years ago

@SabotageAndi I actually created a new project from scratch using the VS extension template.

SabotageAndi commented 2 years ago

in which VS version?

mocsharp commented 2 years ago

2022 Version 17.1.0

SabotageAndi commented 2 years ago

I followed your step but I couldn't reproduce this. Please put somewhere a project where you can reproduce this issue.

mocsharp commented 2 years ago

The sample project can be found on https://github.com/mocsharp/specflow2560

I have updated Steps to reproduce section to use Test Explorer to run the test.

SabotageAndi commented 2 years ago

@mocsharp I still can't reproduce it. I am using the latest version of VS 2022. Just to check, do you have live unit testing enabled?

mocsharp commented 2 years ago

No, I don't. I'm using the latest version of VS Community which doesn't have the live unit test feature.

Microsoft Visual Studio Community 2022 (64-bit) - Current
Version 17.1.0

HairyPorker commented 2 years ago

I experienced exact same behavior. Might worth mentioning that running test using cli dotnet test run once as expected (only call once).

Additionally, I added Unit Test, project and it have the same issue; I use different solution without Specflow project also resulting the same. I think this is an issue from the Test Explorer runner itself ; not specific to Specflow

SpecFlowOSS / SpecFlow