xUnit-Plugin does not generate traits from scenario outline examples

Jazzman82 commented 4 years ago

When using a tagged Examples-Section in a Scenario Outline, the tags are not translated to xunit-category traits and therefore cannot be controlled by the --filter argument of the dotnet test-command.

SpecFlow Version:

[x] 3.1
[ ] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ ] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[x] Xunit

Version number:

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[ ] Classic project format using <PackageReference> tags
[x] Sdk-style project format

.feature.cs files are generated using

[x] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[x] VS 2019
[ ] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[x] Disabled

Are the latest Visual Studio updates installed?

[ ] Yes
[x] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[x] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[x] Command line – PLEASE SPECIFY THE FULL COMMAND LINE dotnet test --filter TestCategory=Something

<SpecFlow> Section in app.config or content of specflow.json

Repro Project

https://github.com/Jazzman82/SpecFlowTagsInScenarioOutline

Issue Description

Steps to Reproduce

In the linked repo, there are projects for MSTest, NUnit and xUnit with the same sources for the .feature and the StepDefinitions.cs. When you run the tests with dotnet test --filter TestCategory=Something then exactly one test should be executed. This works for MSTest and NUnit but not for xUnit - No test is found.

I think this is because in the generated feature.cs-File, the tag is not present on the methods.

Jazzman82 commented 4 years ago

Just for clarification: When using xunit, the command line is expected to be "dotnet test --filter Category=Something" instead of "dotnet test --filter TestCategory=Something" for the other two frameworks.

epresi commented 3 years ago

This is not a bug, but the way how xUnit works. What SpecFlow does: transforms your scenario outline to InlineData: From this:

Scenario Outline: Test filters
    Given a parameter named <name>
    When testrun is started with filter
    Then the correct tests are executed

    @Something
    Examples: 
    |name     |
    |Something|

    @Different
    Examples:
    |name     |
    |different|

to this (auto-generated code):

[Xunit.SkippableTheoryAttribute(DisplayName="Test filters")]
[Xunit.TraitAttribute("Description", "Test filters")]
[Xunit.InlineDataAttribute("Something", new string[] { "Something" })]
[Xunit.InlineDataAttribute("different", new string[] { "Different" })]
public virtual void TestFilters(string name, string[] exampleTags)

Or something like this, if you would write the code:

[Theory]
[InlineData("Something", new string[] { "Something" })]
[InlineData("different", new string[] { "Different" })]
public virtual void TestFilters(string name, string[] exampleTags)

Currently it is not possible to control which InlineData to execute (execute all or none of them, but you cannot apply Trait on the individual InlineData attributes to filter them)

However they are working on this, and hopefully it will be available with xunit3. Check https://github.com/xunit/xunit/issues/1758 for further details.

SpecFlowOSS / SpecFlow