Build.cmd doesn't show failing tests inline

[davkean]

We used to show the actual failed tests inline in the console output so you could quickly see the test that failed, now build.cmd it just tells you the dll:

  [...]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.Un
  itTests.dll [net461|x86]
XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.Unit
  Tests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.CSharp.VS.Unit
  Tests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTests
  .dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTes
  ts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.VS.Uni
  tTests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS.U
  nitTests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS
  .UnitTests.dll [net461|x86]

Build FAILED.

XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
    0 Warning(s)

[natemcmaster]

+1 just ran into this too. This is especially problematic for PRs because you can't get access to the failing test results file and there is no console output. Could we enable an option to show test failures in stdout? - @tmat

[tmat]

Jenkins has UI that shows list of failed tests. Doesn't it?

[jaredpar]

Agree with Tomas. If the issue is lack of test results in PR then that is a separate issue. Both Jenkins and VSTS support xunit result files and will display the failures.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Tomáš Matoušek notifications@github.com Sent: Friday, March 2, 2018 10:23:10 AM To: dotnet/roslyn-tools Cc: Subscribed Subject: Re: [dotnet/roslyn-tools] Build.cmd doesn't show failing tests inline (#82)

Jenkins has UI that shows list of failed tests. Doesn't it?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Froslyn-tools%2Fissues%2F82%23issuecomment-370008234&data=04%7C01%7Cjaredpar%40microsoft.com%7Cb590ecdf7d4b4599591b08d5806aa7af%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636556118194798999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=nH98Bngwh%2BtwYYWeX2ye5TdMtAid%2B94RFc67f7%2Bcm6o%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAI-F7IvdE0AuNBaTW0GvLA6iVQCKrngks5taY4NgaJpZM4QY-EC&data=04%7C01%7Cjaredpar%40microsoft.com%7Cb590ecdf7d4b4599591b08d5806aa7af%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636556118194798999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=0pWlhOc6wMulMCKf%2FEaUv5gb9Sr69TJ16mu%2FowJUWf4%3D&reserved=0.

[natemcmaster]

It does, but other CI services don't, like Travis CI (what we currently use for aspnetcore). Also, in the particular case I just hit, it appears Jenkins isn't grabbing the test file.

[jaredpar]

Can u link to the Jenkins job?

Given the path forward here with our overall infra I think we should be focusing on Jenkins and VSTS.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Nate McMaster notifications@github.com Sent: Friday, March 2, 2018 10:26:44 AM To: dotnet/roslyn-tools Cc: Jared Parsons; Comment Subject: Re: [dotnet/roslyn-tools] Build.cmd doesn't show failing tests inline (#82)

It does, but other CI services don't, like Travis CI (what we currently use for aspnetcore). Also, in the particular case I just hit, it appears Jenkins isn't grabbing the test file.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Froslyn-tools%2Fissues%2F82%23issuecomment-370009674&data=04%7C01%7Cjaredpar%40microsoft.com%7C42d0c780b80f4f67a51508d5806b276f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636556120070328295%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=PKGoOtls3WGtJvuLfAeefoJWZa%2FHWaas1yCdWJH3Tfo%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAI-F1cj1XKEP3WI_dn4CvX_9EmWLVPIks5taY7igaJpZM4QY-EC&data=04%7C01%7Cjaredpar%40microsoft.com%7C42d0c780b80f4f67a51508d5806b276f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636556120070328295%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=xzuO2XvvS3Y0oQufa7LM1cX7TNFBon8hVK6lUKeloAQ%3D&reserved=0.

[natemcmaster]

https://ci.dot.net/job/Private/job/dotnet_arcade/job/master/job/Ubuntu16.04_Debug_prtest/4/

If the Linux failures are the same thing I'm seeing on my macbook, it appears that we're missing a trailing slash on the packages root when launching xunit.runner.console.dll

[davkean]

@jaredpar I run tests via command-line and want to see failures inline without needing to open a html file when I run via command-line.

[jaredpar]

@tmat