Cannot use /TestCaseFilter when running NTVS tests under vstest.console.exe

[ozyx]

Forgive me if this isn't the right place to raise this issue, but I've recently been trying to run some Mocha tests as part of a build step in TFS 2015 and it would be of great help if it were possible to use the 'Test Filter criteria' field to filter Mocha tests, perhaps using Mocha's --grep option.

Nothing that I enter in the 'Test Filter criteria' field seems to change which mocha tests are run, so I figure this feature is not yet implemented.

Any information is greatly appreciated. Thanks!

Expected Behavior

'Test Filter criteria' field filters which Mocha tests are run as part of a build step in TFS 2015.

Actual Behavior

All Mocha tests in NTVS project are run no matter the Test Filter criteria.

• NTVS Version: 1.1
• Visual Studio Version: 2013 Professional Update 5
• Node.js Version: 4.4.7

  Steps to Reproduce

  1. Create NTVS project with Mocha tests
  2. Create build definition with Visual Studio Test step
  3. Enter criteria for Test Filter

[mjbvz]

Thanks.

I haven't tested on TFS directly, but can you provide a list of example test names and some of the criteria you are using? Have you tried something like Name~MyTestName?

Testing is an area that could really be improved (see #494 for example), so if anyone wants to look into fixing this, please let me know. I'm not sure mocha --grep is the best solution since it would be so mocha specific, but we should be able to better integrate these tests with the visual studio testing frameworks.

[ozyx]

Sure. Test names:

DLLExists Test.API.dll
DLLExists SHOULD_NOT_EXIST.dll
VLE Test1_S1
VLE Test1_S2
VLE Test1_S3
VLE Test1_S4
VLE Test1_S5
VLE Test1_S6
VLE Test1_S7

Test filter criteria I've tried:

Name~DLLExists
Name~VLE
Name~zzxyzzxyzz (presumably no tests should run)
Name=VLE Test1_S1
Name~SHOULD
FullyQualifiedName=VLE Test1_S1
...

Nothing seems to affect which tests are selected to run, they all run regardless of the filter. I'd like to +1 #494, implementing trait support and having them be filterable I think would be a worthwhile feature addition.

Thanks for all your hard work. I definitely appreciate the help you've provided.

[mjbvz]

After more testing, the error seems more broad: /TestCaseFilter does not work even when running vstest.console.exe locally:

vstest.console.exe .\MySuperCoolProjectWithLotsOfTests.njsproj /TestCaseFilter:Name=TestThatThisTestDoesNotExist /TestAdapterPath:".\bin\ntvs"

This always runs all tests in the project. No idea if its a bug on our side or with vstest.console.exe. I've updated the issue title to reflect this finding.

[kasajian]

This is unrelated to TFS. The support for Categories / Traits is needed. This is also useful using using Test Explorer

Someone should associate this tag with #494 and close this one.

[paulvanbrenk]

Closing as this is indeed a duplicate of #494 and we need changes in the test explorer/runner/discover to make this work.