search

microsoft / testfx

MSTest framework and adapter
MIT License
742 stars 255 forks source link

--list-tests does not work with --filter #2604

Closed bmcclory closed 6 months ago

bmcclory commented 7 months ago

Describe the bug

Cannot combine --list-tests and --filter options. --list-tests always dumps all tests and ignores filters.

Expected behavior

I'd like to be able to e.g. --list-tests --filter TestCategory=Regression

Additional context

I'm trying to do this in order to achieve test parallelization across machines in Azure DevOps, using a technique like the one described here:

https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-any-test-runner?view=azure-devops https://github.com/idubnori/ParallelTestingSample-dotnet-core

It's a simple algorithm for slicing tests among N agent machines, using the test discovery (--list-tests) output. If there's a better way to accomplish this with the new MSTest runner, please advise.

AB#1999683

Evangelink commented 7 months ago

Hi @bmcclory,

I was convinced this wasn't working with VSTest but it does. Ok so that's a bug on the runner side.

We will work on providing a fix.

testplatform-bot commented 7 months ago

✅ Successfully linked to Azure Boards work item(s):

bmcclory commented 7 months ago

Thanks for looking into this. While I think it's generally useful for --list-tests to honor --filter, I'm also interested if there are other recommended ways of accomplish my ultimate goal here -- to slice a large test suite up for parallel execution across agents in a pipeline execution context.

The VSTest Azure DevOps task offers sophisticated mechanisms for automatically detecting multi-agent parallelism and options slicing test suites by historical execution time.

I'm not familiar with how much of those orchestration behaviors were provided by VSTest runner or the agent task itself, but ultimately I'm chasing a similar solution, but need it to be X-Plat (in addition to other benefits of MSTest runner)

Evangelink commented 7 months ago

The short answer is that there is nothing yet but this is something we already discussed and agree upon that needs to be provided. For sure this isn't part of the current sprint but I am confident we should be able to start back some discussion and design around it after that.