search

dotnet / arcade

Tools that provide common build infrastructure for multiple .NET Foundation projects.
MIT License
672 stars 349 forks source link

Failing tests don't fail build when warningsAsErrors is false #1589

Open ryanbrandenburg opened 5 years ago

ryanbrandenburg commented 5 years ago

We've started setting warningsAsErrors to false in our builds of Arcade projects because that's what we did before, and we have some warnings that would currently cause us to start failing, but when we do this test failures don't fail the build because they're only logged as warnings. I suspect this is caused by this line, where LogStandardErrorAsError="false". I've noticed that that's set that way in a couple different Exec's, so possibly this is not the only area affected.

Repro:

  1. Clone git@github.com:aspnet/Extensions.git
  2. git checkout rybrande/Arcade
  3. .\build.cmd

That takes a while to run, but if I'm right it should replicated against any arcade repo with a failing test against which warnaserrors=false was passed.

ryanbrandenburg commented 5 years ago

Here is an example of this in the wild.

CC @markwilkie

tmat commented 5 years ago

This line should fail the build: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Arcade.Sdk/tools/XUnit/XUnit.targets#L112

ryanbrandenburg commented 5 years ago

That line is output, but as a warning. You can see it at the very bottom of the log I linked above.

natemcmaster commented 5 years ago

+1 to what Ryan said. It's clearly a bug.

image

nguerrera commented 5 years ago

It's behaving as though there's a ContinueOnError=WarnAndContinue in effect when that Error is logged.

nguerrera commented 5 years ago

@rainersigwald

tmat commented 5 years ago

Yeah, seems like a bug in msbuild to me.

markwilkie commented 5 years ago

Next step is to find and reference the tracking msbuild bug (assuming there is one)