search

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration
https://www.specflow.org/
Other
2.22k stars 752 forks source link

Test explorer shows tests without fixture data on performing "Clean" in Visual studio #2635

Closed Saikrishna-Photon closed 1 year ago

Saikrishna-Photon commented 1 year ago

SpecFlow Version

3.9.74

Which test runner are you using?

NUnit

Test Runner Version Number

3.13.3

.NET Implementation

.NET 6.0

Project Format of the SpecFlow project

Classic project format using <PackageReference> tags

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

No response

Issue Description

In our actual project we inherit a hooks class to a partial Feature class and pass on the browser configuration. So in test explorer we see tests with all the browsers passed in the TestFixtures. When we try to perform a "Clean" action in visual studio we see a separate test group along the fixture tests. But when we try to "Build" the project we only see the tests derived from test fixture data after build completion.

Example: The repro project linked to this issues has a feature file with two test cases TC01 and TC02. The test fixture contains information for four browsers like Chrome, Firefox, Edge and Safari. So when we build the project we see four test cases in the test explorer as in the below screenshot.

image

But when we try to clean the solution we see a separate group for two tests TC01 and TC02 along the four test fixture data making a total of ten test cases as in the below screenshot.

image

Steps to Reproduce

Link to Repro Project

https://github.com/Saikrishna-Photon/Sample_TestExplorerIssue

SabotageAndi commented 1 year ago

I had a look at it and I think, we can't do anything to fix it as it looks like an test discovery bug in TestExplorer. Since VS 2019 Visual Studio has buildin test discovery for NUnit, xUnit and MSTest. But it doesn't cover all features from NUnit. And I think you hit such a case.

If you disable "Discover tests in real time from C# ...." in "Options/Test": image the buildin test discovery is disabled.

You only get tests displayed, when the project is built. As clean removes the feature.cs files, the test explorer is empty afterwards.

Btw, I have no idea how SpecFlow behaves with TestFixture attributes. We don't test it.

github-actions[bot] commented 1 year ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.