search

xunit / visualstudio.xunit

VSTest runner for xUnit.net (for Visual Studio Test Explorer and dotnet test)
https://xunit.net/
Other
144 stars 81 forks source link

xUnit test runner doesn't respect parallel option under cog for test playlist #395

Closed framlingham closed 8 months ago

framlingham commented 9 months ago

We have been using this option in VS2022 17.7.6. I think it isn't working for us in our .NET Framework (.NET 4.8) solution for a test playlist. image But after staring at the little spinners, we tried updating our xunit.runner.json:

{
    "longRunningTestSeconds": 600,
    "methodDisplay": "method",
    "methodDisplayOptions": "all",
    "parallelizeAssembly": false,
    "parallelizeTestCollections": false
}

Those last two seem to do it just fine, but I would have expected the VS option to control this too. I tried updating all our xUnit NuGets, but no luck:

framlingham commented 9 months ago

I checked out similar issues, I can't find anything that seems to match or they are a lot older:

bradwilson commented 9 months ago

Can you provide a repro? Is this a single project or multiple projects? What is the behavior you're expecting vs. getting? Which xunit.runner.json options are you changing, and what result does that have?

If you've read through #191 you'd see that we have no control over how multi-assembly parallelism is executed in VSTest. We can instruct people on how to invoke dotnet test vs. dotnet vstest to try to simulate some parallelism options, but it's not clear to me what that menu option is intended to do.

framlingham commented 9 months ago

Also merry Christmas! No rush on this issue since we've worked around it.

framlingham commented 9 months ago

Sorry, I didn't get to your detailed response. So attempting to read it properly:

Behavior dotnet test dotnet vstest VS2022 UI
Run all tests in parallel yes, forced default by processes yes, --parallel and list all assemblies yes, default
Run all tests in series separate commands (per project) yes, list all assemblies done with mouse, one at a time
Run subset nope yes yes

But I get the impression that not much of this is controllable by xUnit or visualstudio.xunit. Who owns what so I can pester the right people/orgs?

framlingham commented 9 months ago

I'll see what I can do about a repro. It is unit tests within same assembly/project. I've tried xunit.runner.json with and without the below two lines:

"parallelizeAssembly": false,
"parallelizeTestCollections": false

It does what it says - what I'd like to try to do is to get the "Run Tests In Parallel" option I screen grabbed earlier to do the same as this. I get the impression that I might need to put this request to another organisation? If so, could you direct me please? I'm not sure who owns what, so please forgive me if I got it wrong.

framlingham commented 9 months ago

OK, so I tested these combinations. The first column mentiones whether the two json lines were included that specify sequential for assembly and test collections. I only ran the one test project/assembly:

xunit.runner.json UI parallel selected Result
with 2 lines parallel Ran sequentially as specified in xunit.runner.json, finished in 5min
with 2 lines sequential Ran sequentially as specified in xunit.runner.json, finished in 5min
without 2 lines parallel Ran parallel as specified in xunit.runner.json, stalled around 5min, stopped a bit later manually
without 2 lines sequential Ran sequentially as specified in xunit.runner.json, stalled around 5min, it aborted at about 9 minutes

It looks like it would be nice if whoever wrote the UI part had some logic that hides the setting (or a hint that an xunit.runner.json is defined) or adds an option to do what the xunit.runner.json says.

bradwilson commented 8 months ago

I don't think there's anything we can do about this.

It's a well known "feature" of VSTest that the way they invoke us completely ignores the parallelizeAssembly setting that we have, so they're completely in control here and we have no say at all.

As for parallelizeTestCollections, that's definitely always in our control and should be independent of whatever you choose with the cog.

Who owns what so I can pester the right people/orgs?

VSTest: https://github.com/microsoft/vstest/

You can bug all you want, but I can pretty much guarantee you they're going to tell you it's working as they want it to work. (Not to say that you shouldn't try, just trying not to get your hopes up.)