microsoft / testfx

MSTest framework and adapter
MIT License
679 stars 246 forks source link

Minimum expected tests at sln/group of projects level #3207

Open Evangelink opened 3 days ago

Evangelink commented 3 days ago

It would be interesting if we were updating some commands to behave at group level instead to be only passed down to each test application. For example, --minimum-expected-tests could be working at global level so that if I filter some projects/namespaces I don't have a failure because some tests were being run.

dotnet test -p:TestingPlatformCommandLineArguments=" --filter FullyQualifiedName!~TestProjectFilter2" .\TestProjectFilter.sln
  Determining projects to restore...
  All projects are up-to-date for restore.
  TestProjectFilter2 -> C:\Users\user\Documents\TestProjectFilter\TestProjectFilter2\bin\Debug\net8.0\TestProjectFilter2.dll
  TestProjectFilter1 -> C:\Users\user\Documents\TestProjectFilter\TestProjectFilter1\bin\Debug\net8.0\TestProjectFilter1.dll
  Run tests: 'C:\Users\user\Documents\TestProjectFilter\TestProjectFilter1\bin\Debug\net8.0\TestProjectFilter1.dll' [net8.0|x64]
  Run tests: 'C:\Users\user\Documents\TestProjectFilter\TestProjectFilter2\bin\Debug\net8.0\TestProjectFilter2.dll' [net8.0|x64]
  Passed! - Failed: 0, Passed: 0, Skipped: 0, Total: 0, Duration: 87ms
  Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 112ms
C:\Users\user\Documents\TestProjectFilter\TestProjectFilter2\bin\Debug\net8.0\TestProjectFilter2.dll : error run failed: Tests failed: 'C:\Users\user\Documents\TestProjectFilter\TestProjec
tFilter2\bin\Debug\net8.0\TestResults\TestProjectFilter2_net8.0_x64.log'

TestProjectFilter.zip

As you can see, it is not possible to filter out the whole project without adding --ignore-exit-code 8

Originally posted by @SymbioticKilla in https://github.com/microsoft/testfx/issues/3121#issuecomment-2203255697