Execute and publish xUnit Tests results with .NET Core

[rschiefer]

I'm struggling to get my xUnit tests from a .NET Core app executing and the results published to my VSTS build. What's the trick?

I've tried the VS Test task with the xUnit test adapter package restored at the solution level. The tests aren't found.

I've tried running "dotnet test" and publishing test results but they aren't found.

I likely missed some setup on one of these but need help at this point.

[atinb]

Did you try providing a path to the Test Adapter package when using VSTest task? By default, we look under the packages folder in the solution directory. When you tried just publishing results, were the tests not found or the tests were executed but results file was not found?

[rschiefer]

My runner is being restored via Nuget and placed in the packages folder in the solution directory. I also set the adapter path manually to $(Build.SourcesDirectory)\packages but neither ran my tests.

"dotnet test" does run my tests but publish could not find the test output.

I would prefer to get the VS Test task working since it supports code coverage.

[atinb]

Robb, yes, you cannot use VSTest task currently to run .NET Core tests. "dotnet test" is the way to run them. You should be able to use Publish test results task to upload the test results. Please let us know the error you are seeing. Most likely the path to the result directory is incorrect.

For CC, the only tool that supports .NET Core is Open Cover. Using that and the OpenCover to Cobertura convertor, you can publish CC data also using the Publish Code Coverage task. Thanks!

[rschiefer]

Others have had success using the VS Test task, but I can't get it working - https://github.com/xunit/xunit/issues/895

Regarding publishing dotnet test results, I don't get an error it just doesn't find any results.

What should I set the results directory to? Have you seen a tutorial/blog post for setting up Open Cover?

[rschiefer]

still no luck on my end. any further ideas?

[atinb]

@rschiefer My understanding of the other thread is that they are not running CoreCLR tests. Are you running tests on the hosted agent or local? If local, can you verify that the place where you are pointing for the test files actually exists? Can you also attach the logs for the Publish task? Thanks!

[rschiefer]

I used this tutorial - https://xunit.github.io/docs/getting-started-dotnet-core.html The tests run locally and will also run via a dotnet test command in a VSTS build. I tried to publish test results in the VSTS build but it couldn't find the results.

Here is the test run output:

D:\Github\ConCode.NET\src\ConCode.NET.Tests>dotnet test Project ConCode.NET.Core (.NETStandard,Version=v1.6) was previously compiled. Skipping compilation. Project ConCode.NET.Web (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation. Project ConCode.NET.Tests (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation. xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64) Discovering: ConCode.NET.Tests Discovered: ConCode.NET.Tests Starting: ConCode.NET.Tests Finished: ConCode.NET.Tests === TEST EXECUTION SUMMARY === ConCode.NET.Tests Total: 21, Errors: 0, Failed: 0, Skipped: 0, Time: 2.518s SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

[atinb]

We will need the logs of the Publish test results task. Kindly attach that. Thanks!

[Sebosek]

I think, there is misunderstanding between you guys, @rschiefer just want to know, how to setup tests for .net core on vso. It's actually pretty easy, you need 3 tasks. First, Command Line, with tool dotnet and arguments restore -s https://api.nuget.org/v3/index.json Second, Command Line, with tool dotnet and arguments test -xml ./TEST-idea.xml with working folder setup to your path to tests Third, Publish Test Results, with Test Result Format XUnit and that's it.

Hope it helps. S.

[rschiefer]

YES! Thank you @Sebosek! I was missing the -XML flag.

[atinb]

Thanks @Sebosek Sorry for the misunderstanding @rschiefer

[vanthome]

@rschiefer did you get it to work with the VS Test task? I want to achieve the same as I think it's more convenient compared to specifiying dotnet test etc. manually.

[rschiefer]

No. I agree. Hope that's in the works.

[bryanmacfarlane]

@atinb - can you work with Andy to ensure this is documented?

[Allann]

To get vstest working I had to do the following:

• In NuGet restore task - add -PackagesDirectory $(Build.SourcesDirectory)\packages\ in NuGet arguments
• In VSTest task - add $(Build.SourcesDirectory)\packages\ to Advanced Execution Options 'Path to Custom Test Adapter'.

Hope that helps.

[saoc90]

@Allann How does your project.json look like? Because for me its just says : specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.. And i've checked it the packages are in this folder.

[Allann]

Test project

{
  "dependencies": {
    "ModuleInit.Fody": "1.5.9",
    "Moq": "4.5.21",
    "Newtonsoft.Json": "9.0.1",
    "xunit": "2.1.0",
    "xunit.runner.visualstudio": "2.1.0"
  },
  "frameworks": {
    "net461": {}
  },
  "runtimes": {
    "win-anycpu": {},
    "win": {}
  }
}

[tiefling]

Did you get any decent visible output with xUnit on VSTS in the end using the dotnet tasks to run and the publish task on the output files? I've been trying to get NUnit working and that's a flat no on any kind of visual output at all.

[rschiefer]

You get some basic pass/fail details:

[mattbrooks2010]

Getting quite a bit more detail than that using TFS 2015.3 on-premise.

Note that the 'Outcome' filter defaults to 'Failed' for some reason. This seems like a bug.

[atinb]

@rschiefer You need to change the filter to Outcome = All as @mattbrooks2010 showed above @mattbrooks2010 The idea was to show the failing tests first as the call to action. @tiefling We are working on adding support for NUnit 3.0 result format

[mattbrooks2010]

@atinb thanks for the insight. That does make sense but I would perhaps suggest, as a usability enhancement, that if there are no failed tests then it would be better to default to one of the other options.

[rschiefer]

Yep, forgot about that option. Thanks @atinb!

[Thaoden]

I'm struggling with getting my xUnit tests to run on my TFS. I added the packages paths to the NuGet restore, dotnet restore and VSTest tasks as stated above. When I run the build process, my tests are not discovered (although the test assemblies are found).

When I use a command line task with dotnet test --no-build -xml ./result.xml with working directory set to one of the test projects, it fails with

No executable found matching command "dotnet-test-xunit"

, which is normal I guess, as I don't have the runner in my test project.

I then modified my project.json file so that it matches the above version by addigd a reference to xunit.runner.visualstudio, the build solution task failed with

[error]MyTestProject\project.json(8,40): Error NU1001: The dependency xunit.runner.visualstudio >= 2.1.0 could not be resolved.

So what am I missing?

[tiefling]

Check out the end of this Stack Overflow post I wrote for directions on how to get this all working - there's some links there http://stackoverflow.com/q/40093573/2579219 Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Thaoden notifications@github.com Sent: Wednesday, October 19, 2016 10:16:36 AM To: Microsoft/vsts-tasks Cc: Keith Jackson; Mention Subject: Re: [Microsoft/vsts-tasks] Execute and publish xUnit Tests results with .NET Core (#2126)

I'm struggling with getting my xUnit tests to run on my TFS. I added the packages paths to the NuGet restore, dotnet restore and VSTest tasks as stated above. When I run the build process, my tests are not discovered (although the test assemblies are found).

When I use a command line task with dotnet test --no-build -xml ./result.xml with working directory set to one of the test projects, it fails with

No executable found matching command "dotnet-test-xunit"

, which is normal I guess, as I don't have the runner in my test project.

I then modified my project.json file so that it matches the above version by addigd a reference to xunit.runner.visualstudio, the build solution task failed with

[error]MyTestProject\project.json(8,40): Error NU1001: The dependency xunit.runner.visualstudio >= 2.1.0 could not be resolved.

So what am I missing?

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-tasks/issues/2126#issuecomment-254759021, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABvhYXLXFmPZvovm_-qPmyCPx0XWlH7dks5q1d_0gaJpZM4JTfeY.

[Thaoden]

@tiefling Thanks for the link. I tried but this doesn't work for me as I have an ASP.Net Core app running on full .Net framework as my system under test. Some of the references of the web project cannot be resolved:

The dependency Microsoft.AspNet.WebApi.Core 5.2.3 does not support framework .NETCoreApp,Version=v1.0.

[tiefling]

Try changing the imports or framework dependency in the test project to match and see how you go.

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Thaoden notifications@github.com Sent: Wednesday, October 19, 2016 12:38:05 PM To: Microsoft/vsts-tasks Cc: Keith Jackson; Mention Subject: Re: [Microsoft/vsts-tasks] Execute and publish xUnit Tests results with .NET Core (#2126)

@tieflinghttps://github.com/tiefling Thanks for the link. I tried but this doesn't work for me as I have an ASP.Net Core app running on full .Net framework as my system under test. Some of the references of the web project cannot be resolved:

The dependency Microsoft.AspNet.WebApi.Core 5.2.3 does not support framework .NETCoreApp,Version=v1.0.

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Microsoft/vsts-tasks/issues/2126#issuecomment-254788004, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABvhYRyXiZB8B-5mwip4uUyAK1UbmOmFks5q1gEdgaJpZM4JTfeY.

[Thaoden]

@tiefling To be able to reference my main projects I have to set the test project framework to net461 and then I'm back at the beginning.

[skorunka]

What is a recommended way to run xunit tests for asp.net core in VSO? is it a ".NET Core(Preview)" task(using dotnet test) or "Visual Studio Test" task? The 1st works fine except: http://stackoverflow.com/questions/40641430/generate-test-results-using-xunit-in-vso-build-task-for-asp-net-core-app

[tiefling]

I set mine up using simple command line tasks...

Tool: dotnet Arguments: test -xml testresults.xml Working folder: {insert the folder for the project to test here}

The .Net Core options are new - Not sure how they differ but the absence of a working directory may mean that the test results file is being generated at the solution root each time

[StrangeWill]

The only thing that is rough is I don't think we can get test coverage if we don't use a Visual Studio Test task, however I'm not entirely sure that would be supported on .NET Core anyway (TeamCity has been struggling off and on, they had it working during RC2 but haven't since).

Right now I'm just using dotnet test -xml results.xml and having VSTS suck in the XML results, nice, but no code coverage, and a solution with multiple test projects require multiple commands right now.

[raffaeu]

@StrangeWill that's the same problem I am facing. Working on it and trying to see if I can create some sort of recursive task to run all my test projects and output everything into one .xml file and then give it back to TFS online to get a full list of executed tests

[raffaeu]

After few days of struggle I came up with probably the most feasible solution. I have multiple projects and multiple test projects and I use xunit with Dot Net Core.

Restore References First I restore my references by iterating all available projects

1. Step .NET Core and set the following: 1.1. Command: Restore 1.2. Projects: **/project.json

Build everything Then I build everything using dotnet build

2. Step .NET Core and set the following: 2.1. Command: Build 2.2. Projects: **/project.json

Run tests Then I run my tests using dotnet test together with PShell

3. Step Power Shell and set the following: 3.1. Type : Inline Script 3.2. Script: dir test | % { dotnet test $_.FullName -xml "results/$_.results.xml" }

Publish Publish all tests results

4. Final add a step Publish Test Results 4.1. Format: XUnit 4.2. File: **/*result*.xml

Note Important is the step 3. For me, the test folder is named test but you should define your own there.

Next Now I will try to figure out how to run Code Coverage

[StrangeWill]

@raffaeu definitely a nice solution to handle any number of test projects -- but won't get us code coverage without VSTest task support. I may be reaching for something that is no supported to begin with though, but I'd really like code coverage reports if Microsoft supports that on .NET Core.

[raffaeu]

@StrangeWill I will make more investigations using Microsoft.CodeCoverage which is available here: Nuget I just need some time to experiment it and I should come up with a solution here.

[Thaoden]

@raffaeu Which version of TFS are you running this on?

[raffaeu]

@Thaoden I do use VSTS (Visual Studio Team Service) which is the online version of TFS. I think is the version 2017 right now but I really have no idea because VSTS keeps updating itself almost every week.

[StefH]

@tiefling

Did you solve that No executable found matching command "dotnet-test-xunit" issue ? Because I can run nunit tests like this:

OpenCover\tools\OpenCover.Console.exe -register:user -target:"nunit3-console.exe"

But how to run xUnit ?

[Jonathan34]

vs.test.console.exe should support .net core 1.0 and 1.1, i hope it is coming soon, anyone knows?

[raffaeu]

So, yesterday they announced VS 2017 and the new VSTS but if I am looking online at the build editor, nothing changed yet. Any news?

[raffaeu]

Oh come'n guys now you removed the -xml results.xml switch?? Between 1.0 to 1.1 version of dotnet CLI? Do you have Product Owner/Agile Process to follow in MS?

[Jonathan34]

@raffaeu you can use -l trx

[oatsoda]

For anyone reading this, as per @Jonathan34 comment, this is the full command if you want to name your output file: -l trx;logfilename=testresults.xml

And you need to remember to change your Publish Test Results task to VSTest format, not xUnit (or whatever you are actually using for your tests). Presumably the trx logger is fixed on the VSTest format.

[plaisted]

In case anyone is struggling with getting Xunit test results to show up in VSO using the new csproj/dotnet tooling I've outlined the required steps below. These steps are all combining the suggestions above and adding a few things.

1) Add the "xunit.runner.visualstudio" nuget package to your test project. This package allows the "dotnet test" command to find your tests. Below is an example from a XUnit test project of mine:

2) Set your build to run under the VS2017 hosted agent type or use self hosted with .csproj dotnet core tooling installed (under build options):

3) Add a dotnet restore step for your project or all csproj fies (eg. below), this may not be necessary as dotnet test may restore if needed:

4) Add a "Inline" powershell step to run the VSTest runner for your test projects (note you may need to change your script for your naming convention, I always end my projects as seen below):

gci -rec `
| ? { $_.Name -like "*.IntegrationTests.csproj" `
       -Or $_.Name -like "*.Test.csproj" `
       -Or $_.Name -like "*.Tests.csproj" `
     } `
| % { dotnet test $_.FullName -l trx`;logfilename=Test_Results.xml }

(multiline powershell for readability)

5) Add a "Publish Test Results" step with VSTest result type:

Should result in tests showing up as before. Note this doesn't produce code coverage, I haven't explored that yet.

(2017-5-20 Updated for hosted agents / package reference information)

[ccit-spence]

Is there a way to do this yet? I have tried the ideas outlined above. Still no results. I would like Code Coverage as well. I have setup a project using Spring Boot and it has test results and code coverage. Can't figure out how to get the same thing to work for ASP.NET Core 1.1

[StrangeWill]

@ccit-spence we just moved off of VSTS, TeamCity has code-coverage tools built in that works with Core and their build system is 100x better.

[ccit-spence]

@StrangeWill Thanks for the info, we are hoping to stay within the MS eco system. We are mainly a Java shop and no stranger to Jetbrains products. Looking at converting some projects to ASP.NET Core. This problem was discovered within the first few days and has made us put things on hold for a bit.

With the switch from project.json and other problems, seems things need to mature a little more. Especially documentation.

[ericsciple]

@StrangeWill I'm very interested in the build system feedback. If you email ersciple the feedback (at microsoft com), I can distribute to the appropriate folks.

[plaisted]

@ccit-spence I've updated my comment above to provide additional information for getting the test results to appear using hosted agents. Note there is still no code coverage but I just testing it using a hosted agent and fresh build and works for getting test results (for me).

[brunobertechini]

Just would like to know if someone can point me to any resolution to this problem (related to this thread):

I have dotnet test running fine on my achine using xunit. Couple of my xunit tests have the [Trait("Type", "Integration")] decorator.

In my developer machine I run dotnet test like this:

dotnet test -c release --filter "Type!=Integration" path-to-my-test-project.csproj

the command runs fine and ignores the decorated (integration) tests.

When I run in VSO/VSTS I get this message:

Exception filtering tests: No tests matched the filter because it contains one or more properties that are not valid (Type). Specify filter expression containing valid properties (DisplayName, FullyQualifiedName) and try again.

My local dotnet --version = 1.0.4 My local dotnet --info :

.NET Command Line Tools (1.0.4)

Product Information:
 Version:            1.0.4
 Commit SHA-1 hash:  af1e6684fd

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.15063
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.4