Azure Devops - Test filter criteria does not work when we use "!=" to exclude certain TestCategory

hawisback commented 4 years ago

Test Filter Criteria does not exclude tests when run in pipeline.

SpecFlow Version:

[x ] 3.1
[ ] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ x] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[ ] Xunit

Version number:

Project Format of the SpecFlow project

[x ] Classic project format using packages.config
[ ] Classic project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[ x] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[x ] VS 2019
[ ] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[x ] Disabled

Are the latest Visual Studio updates installed?

[ x] Yes
[ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[x ] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ x] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK : Task name: Visual Studio Test
[ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config or content of specflow.json

Adding just the configuration section:
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>

Repro Project

Cannot share the project as it is hosted on secured server. ### Issue Description

My test build runs perfectly fine and there are no issues with that. When running in pipeline, we are now having to exclude some tests due to some dependencies and we got this "Test filter criteria" option in Task "Visual Studio Test". The Test filter criteria works fine when we use "=" IS EQUAL and executes the tests with traits that satisfy the condition. While the same fails to exclude tests when using "!=" for NOT condition.

Examples: Works fine: “TestCategory=automated” – executes the scenarios marked with @automated

Does not work fine: Expected: “TestCategory!=Manual” – executes the scenarios that are not marked with @Manual Actual: “TestCategory!=Manual” – executes all the scenarios including the ones that are marked with @Manual

Steps to Reproduce

Create a build. Set the Test filter criteria in Visual Studio Test with below filter criteria. TestCategory!=Manual Expected: Executes the scenarios that are not marked with @Manual

epresi commented 4 years ago

I created a small repro project, but sadly I was not able to reproduce this with the latest SpecFlow and SpecFlow+ Runner. It would be helpful, if you could send us a small repro project.

We could then try the test category filters locally, e.g.: vstest.console.exe "C:\Temp\TestProject\bin\Debug\Tests.dll" /TestCaseFilter:"TestCategory!=Manual"

Also, can you please share your package versions with us? Do you have an srprofile for the Runner? If yes, it could be helpful if you share that too with us.

SabotageAndi commented 3 years ago

Closed because of inactivity

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

SpecFlowOSS / SpecFlow