Closed tteguayco closed 10 months ago
What versions of the NuGet packages are you using? Can you provide a repro project?
Here you can see me illustrate the feature working: https://github.com/xunit/visualstudio.xunit/issues/189#issuecomment-1650259737
With this unit test:
using Xunit;
namespace Empty;
public class UnitTest
{
[Fact] public void Passing1() { }
[Fact] public void Passing2() { }
[Fact] public void Passing3() { }
[Fact] public void Passing4() { }
[Fact] public void Failing1() => Assert.Fail("I'm a failing test");
[Fact] public void Passing5() { }
[Fact] public void Passing6() { }
[Fact] public void Passing7() { }
[Fact] public void Passing8() { }
[Fact] public void Failing2() => Assert.Fail("I'm a failing test");
[Fact] public void Passing9() { }
[Fact] public void Passing10() { }
[Fact] public void Passing11() { }
[Fact] public void Passing12() { }
[Fact] public void Failing3() => Assert.Fail("I'm a failing test");
[Fact] public void Passing13() { }
[Fact] public void Passing14() { }
[Fact] public void Passing15() { }
[Fact] public void Passing16() { }
[Fact] public void Failing4() => Assert.Fail("I'm a failing test");
[Fact] public void Passing17() { }
[Fact] public void Passing18() { }
[Fact] public void Passing19() { }
[Fact] public void Passing20() { }
[Fact] public void Failing5() => Assert.Fail("I'm a failing test");
}
and this project:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<LangVersion>11.0</LangVersion>
<Nullable>enable</Nullable>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
<PackageReference Include="xunit" Version="2.6.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" />
</ItemGroup>
</Project>
dotnet test
:
All projects are up-to-date for restore.
empty -> C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll
Test run for C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.2 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.13] Empty.UnitTest.Failing2 [FAIL]
[xUnit.net 00:00:00.13] Empty.UnitTest.Failing1 [FAIL]
Failed Empty.UnitTest.Failing2 [2 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing2() in C:\Dev\repro\empty\UnitTest.cs:line 16
Failed Empty.UnitTest.Failing1 [< 1 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing1() in C:\Dev\repro\empty\UnitTest.cs:line 11
[xUnit.net 00:00:00.15] Empty.UnitTest.Failing5 [FAIL]
Failed Empty.UnitTest.Failing5 [< 1 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing5() in C:\Dev\repro\empty\UnitTest.cs:line 31
[xUnit.net 00:00:00.15] Empty.UnitTest.Failing4 [FAIL]
[xUnit.net 00:00:00.15] Empty.UnitTest.Failing3 [FAIL]
Failed Empty.UnitTest.Failing4 [< 1 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing4() in C:\Dev\repro\empty\UnitTest.cs:line 26
Failed Empty.UnitTest.Failing3 [< 1 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing3() in C:\Dev\repro\empty\UnitTest.cs:line 21
Failed! - Failed: 5, Passed: 20, Skipped: 0, Total: 25, Duration: 22 ms - empty.dll (net6.0)
dotnet test -- xUnit.StopOnFail=true
:
Determining projects to restore...
Restored C:\Dev\repro\empty\empty.csproj (in 261 ms).
empty -> C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll
Test run for C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.2 (x64)
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.13] Empty.UnitTest.Failing2 [FAIL]
Failed Empty.UnitTest.Failing2 [2 ms]
Error Message:
I'm a failing test
Stack Trace:
at Empty.UnitTest.Failing2() in C:\Dev\repro\empty\UnitTest.cs:line 16
Failed! - Failed: 1, Passed: 0, Skipped: 0, Total: 1, Duration: < 1 ms - empty.dll (net6.0)
Thank you for the detailed answers. I'll have a look as soon as I get a chance.
Ok, I have created this dummy test repo. It has the same NuGet packages you @bradwilson depicted above.
Before adding the file xunit.runner.json
to the root of the project, if I run dotnet test -- xUnit.StopOnFail=true
, everything runs as expected: the execution of the tests stop when a failing test is encountered.
However, if then I add the xunit.runner.json
file according to these instructions here, it doesn't matter if I run dotnet run
or dotnet run -- xUnit.StopOnFail=true
, all tests are run ignoring the property "stopOnFail": true
in the JSON file.
Any ideas?
lol. It looks like setting parallelizeTestCollections: false
makes the config stopOnFail
to be ignored.
Is that the expected behavior?
Also, running the tests from VisualStudio 2022 (right click on project > Run Tests) seems to ignore the config set in xunit.runner.json
.
It looks like setting parallelizeTestCollections: false makes the config stopOnFail to be ignored.
Is that the expected behavior?
Definitely not. That seems like the bug. 👍🏼
Have narrowed down that this appears to be a problem unique to xunit.runner.visualstudio
, as the console runner w/ .NET Framework works as expected:
As well as with .NET:
This is a core framework bug, despite only being visible in the VSTest runner, because the bug is in the synchronous message bus (which gets enabled by VsTestRunner when parallel test collections are turned off):
And we can see that stop-on-fail was never added to the SynchronousMessageBus, as evidenced by the creation here:
Because this is a core framework bug, getting the fix means updating your xunit
reference, and not your xunit.runner.visualstudio
reference.
Available in v2: 2.6.2-pre.10
Available in v3: 0.1.1-pre.321
Thank you for kind assistance. It looks like we need for now parallelizeTestCollections
set to true
, so hopefully this bug will not be affecting us for now.
Again, thank you @bradwilson
I have a project with some xunit tests which includes
xunit.runner.visualstudio
as a dependency (targeting version 2.5.1).Inside that project, I added an
xunit.runner.json
file with the following content:When running my tests from VisualStudio 2022, the option
parallelizeTestCollections
seems to be working fine butstopOnFail
is completely ignored; even if one test fails, all the remaining will run (even if the latter started after the failing one finished).Am I missing something here? Happy to provide more context as necessary.