XUnit2 fails using XUnit 2.3.0

[johnkors]

What You Are Seeing?

Running

        XUnit2(testAssemblies, new XUnit2Settings {});

=>

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in C:\c
ode\frontend\commission\source\Commission.Data.Write.IntegrationTests\bin\release\netcoreapp2.0
An error occurred when executing task 'Test'.
Error: xUnit.net (v2): Process returned an error (exit code 1).

What is Expected?

The same results as either dotnet test or dotnet xunit.

What version of Cake are you using?

<packages>
    <package id="Cake" version="0.22.2" />
</packages>

Are you running on a 32 or 64 bit system?

64

What environment are you running on? Windows? Linux? Mac?

Windows

Are you running on a CI Server? If so, which one?

How Did You Get This To Happen? (Steps to Reproduce)

Run a Xunit test using XUnit 2.3.0 using the XUnit2 Cake runner.

Output Log

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
System.InvalidOperationException: Unknown test framework: could not find xunit.dll (v1) or xunit.execution.*.dll (v2) in C:\c
ode\frontend\commission\source\Commission.Data.Write.IntegrationTests\bin\release\netcoreapp2.0
An error occurred when executing task 'Test'.
Error: xUnit.net (v2): Process returned an error (exit code 1).

[devlead]

XUnit for .NET Core 2.0 isn't compatible with previous versions of the runner.

To achieve same as dotnet test there's the DotNetCoreTest alias.

There's currently no "DotNetCoreXunit" alias (that could be a good additon to Cake though), but there's a DotNetCoreTool alias. So to execute the .NET Core tool xunit there's the DotNetCoreTool alias example:

    var projects = GetFiles("./src/**/*.Tests.csproj");
    foreach(var project in projects)
    {
        DotNetCoreTool(project,
            "xunit",  "--no-build -noshadow -configuration " + parameters.Configuration);
    }

[johnkors]

Ah, okay ! I obviously did not read the docs well enough to realize that XUnit2(testAssemblies, new XUnit2Settings {}); was running it thru an older incompatible console runner. As I remember, it worked fine with the beta packages of xunit2.3.0, so still a bit strange.

Anyhow. I ended up running it thru the DotNetCoreTest as you mention here. It works well enough in my scenario.

Thanks again!

[devlead]

Excellent 👍

[Aleksanderis]

Not sure if this issue was solved correctly. Can you clarify please and re-open if needed?

There is xUnit console runner for netcoreapp1.0 and netcoreapp2.0, it's just not published as *.exe app. Yes, it's relatively new, but it's included in xUnit 2.3.0. If you add a reference to #tool nuget:?package=xunit.runner.console you can find it somewhere in CakeDir\tools\xunit.runner.console\xunit.runner.console\tools\netcoreapp2.0\xunit.console.dll.

It can be executed using dotnet xunit.console.dll (it's not same as dotnet xunit CLI alias, note 'assemblyFiles` argument in the screenshot):

It can be executed as StartProcess of course, but probably could be wrapped perfectly well as proper Cake alias.

dotnet text and dotnext xunit - also good and viable options, but the main problem with them, that they cannot execute tests from multiple assemblies as it's possible with "raw" xUnit. Executing all assemblies separately - is much slower, and reporting is also not perfect if you speak about CI server case.

[devlead]

A proper .NET Core Xunit alias would be a good addition at this point, or flag on xunit which xunit assembly to use.

When executing .NET Core assembles DotNetCoreExecute(FilePath, ProcessArgumentBuilder) is a better option as it handles .NET Core runtime resolution and checks exit code.

[paul42]

hey @devlead as a new cake user running into this, quick q: are there any examples using DotNetCoreExecute() or in your previous example here: https://github.com/cake-build/cake/issues/1872#issuecomment-335699924 is there a fully formed parameters.configuration I should look for?

I'd love to help out if possible! is there a way to edit the documentation to show that Xunit2() alias is not currently selecting the proper xunit assembly?

in your opinion should I just stick to the dotnetcoretest alias and pass xunit info into that?

I love the CAKE tool and ecosystem, huge fan of everything :D