Scenario Outline can not be run with just a single parameterization

SpecFlowOSS / SpecFlow

#1 .NET BDD Framework. SpecFlow automates your testing & works with your existing code. Find Bugs before they happen. Behavior Driven Development helps developers, testers, and business representatives to get a better understanding of their collaboration

https://www.specflow.org/

Other

2.24k stars 753 forks source link

Scenario Outline can not be run with just a single parameterization #1251

Open StarWars999123 opened 6 years ago

StarWars999123 commented 6 years ago

SpecFlow Version:

[ ] 2.4
[x] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ ] SpecFlow+Runner
[ ] MSTest
[x] NUnit
[ ] Xunit

Version number:

Visual Studio Version

[x] VS 2017
[ ] VS 2015
[ ] VS 2013

Are the latest Visual Studio updates installed?

[x] Yes
[ ] No

.NET Framework:

[x] >= .NET 4
[ ] before .NET 4

Test Execution Method:

[x] Visual Studio Test Explorer
[ ] TFS/VSTS – Task – PLEASE SPECIFY THE NAME OF THE TASK
[ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

Issue Description

In older versions, it was possible to execute a single parameterization of a scenario outline. When you try to execute a single parameterization now, it allways executes the complete scenario! However, we have scenarios with dozens of parameters and each test can take several minutes. So, when a single parameter fails, its very time consuming to debug it.

Steps to Reproduce

Have a project, with a scenario outline and several parameters. Run a simple parameter combination from the test explorer. As a result, the complete scenario outline with all parameters get executed.

StarWars999123 commented 6 years ago

It seems, that using Specflow 2.4.0 does not cause this error. However, we got trouble with the newest VS release and Specflow 2.3.0.

StarWars999123 commented 6 years ago

Issue reoccured with Specflow 2.4.0. Its very difficult now, to debug failing tests.

SabotageAndi commented 6 years ago

@StarWars999123 I can't reproduce the behaviour you are describing. I have here an example solution with all test runners: https://github.com/SabotageAndi/SpecFlow.Example.AllTestRunners And executing a single example of a scenario outline is possible.

StarWars999123 commented 5 years ago

Okay, I had a test, where it occured allways, that all parametrizations were run. The issue disappeared, as soon as I added a unique column as a first column and I am now able to debug also a single parametrization.

Scenario Outline: Asdfsdfsdfsdfsdfsdfsd sdfsdfsdf sdff sdfsdfsdfsdfsdf

Gvien I set the settings to 
| Firstvalue|SecondValue|
| <Tz>| <dST>|
When something 
Then something shall be so.

The examples table before:

| Tz       | dST             |
| -18000   | true            |
| 3600     | false           |
| -18000   | false           |

The table afterwards:

| No | Tz | dST |
| 1  | -18000   | true               |
| 2  | 3600     | false              |
| 3  | -18000   | false              |

SabotageAndi commented 5 years ago

So did you solved your problem on your own?

StarWars999123 commented 5 years ago

Its more kind of a workaround. I changed the arguments of the scenario to identify the scenarios faster. However, this seems still kind of weird and shouldn't happen.

Iamgroot27 commented 4 years ago

Hi , Even i am facing this problem. and am using specflow 3.0. Like StarWars999123 mentioned if the first set of parameters in scenario outline is replaced with a unique set , i am able to run specific selected scenarios, if that is not a unique set, then my whole set of possibilities run even if i select only one in test explorer.