[vstest.console][NUnit3Adapter] Using Category as TestCaseFilter throws error

[abhishekkumawat23]

Hello,

We are Automated Testing team from Microsoft.

Issue: We observed that in NUnit3.TestAdapter, Category can't be given as TestCaseFilter.

Repro steps:

1. Refer attached project for sample NUnit project.
2. In sample project, create a NUnitTest project. Write some tests in it. For few of the tests add Category("Manual")
3. Run following command to run tests: vstest.console.exe /TestAdapterPath: /TestCaseFilter:Category!=Manual

Observations: Error: An exception occurred while invoking executor 'executor://nunit3testexecutor/': No tests matched the filter because it contains one or more properties that are not valid (Category). Specify filter expression containing valid properties (FullyQualifiedName, Name, Priority, TestCategory) and try again.

Expected Behavior: As in source code, Category is the triat (and not TestCategory), it is expected to pass Category as TestCaseFilter. Error should not be thrown for Category TestCaseFilter and should be respected.

[abhishekkumawat23]

Note: Please change attachment extension from pdf to zip (I was facing some issue with attaching files with zip extension. Thus changed zip extension to pdf) NUnitTestProject1.pdf

[CharliePoole]

The NUnit3 adapter does not parse the filter expression used in the TestCaseFilter option. This is done by vstest.console, which passes us a list of tests to execute. Hence, you have to use the syntax recognized by vstest, which is TestCategory.

[OsirisTerje]

Just to enhance what @CharliePoole said: See post https://blogs.msdn.microsoft.com/devops/2013/06/11/part-3-unit-testing-with-traits-and-code-coverage-in-visual-studio-2012-using-the-tfs-build-and-the-new-nuget-adapter-approach/
In there, see table: VSTest name: TestCategory == NUnit : Category == MSTest TestCategory

[CharliePoole]

See also nunit/nunit3-vs-adapter#47 , which discusses the possible alternative ways of handling these filters, including parsing the expression ourselves.

BTW, this issue should actually be in the nunit3-vs-adapter repository rather than this one. I won't bother moving it, since we are closing, but please use that repo for future issues against the nunit 3 adapter.

[abhishekkumawat23]

@CharliePoole @OsirisTerje : In https://github.com/nunit/nunit3-vs-adapter/blob/18d0d2bab627e6c568bc34482e5a621133176df9/src/NUnitTestAdapter/TFSTestFilter.cs, NUnit is specifically mapping TestCategory property to Category trait in SupportedProperties.

Unlike mstest, vstest.console understands all syntaxs as defined in SupportedProperties. So if Category property is mapped to Category trait in SupportedProperties, vstest.console will understand Category as TestFilter.

As Nunit has Category as trait, Category as TestFilter property should be supported in addition to TestCategory Property. Our TIA feature is impacted because of Category not being supported as TestFilter property.

Sorry, issue is raised in incorrect repo. Please let me know if I need to move this issue to nunit3-vs-adapter repo.

[OsirisTerje]

1) Yes, issue should be in the nunit3-testadapter repo. 2) In which version of vstest did it support all properties? Afaik it didnt do so in TFS2012.
3) Will older versions accept this trait in both VS and TFS ? 4) If we add the same trait as both TestCategory and Category, it will appear twice in the category list in Test Explorer, which is not a nice experience.

And to let us understand this: 5)MSTest is using TestCategory, is that no longer supported by TIA either? Or has MSTest changed and using Category too?

[abhishekkumawat23]

@OsirisTerje We are discussing it with Unit Testing team. I will create the issue in nunit3-testadapter along with answer to your queries shortly.