search

dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
266 stars 132 forks source link

UX when there is failing PoisonTests in CI could be improved. #3910

Open MichaelSimons opened 9 months ago

MichaelSimons commented 9 months ago

The following information is all that appears when there is a failing poison test in CI

Error message
Assert.Null() Failure: Value is not null
Expected: null
Actual:   "\nExpected file '/vmr/test/Microsoft.DotNet.Source"···

Stack trace
[at Microsoft.DotNet.SourceBuild.SmokeTests.BaselineHelper.CompareFiles(String expectedFilePath, String actualFilePath, ITestOutputHelper outputHelper, Boolean warnOnDiffs) in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs:line 76](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FBaselineHelper.cs&version=GBmain&_a=contents&line=76&lineEnd=77&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.DotNet.SourceBuild.SmokeTests.BaselineHelper.CompareBaselineContents(String baselineFileName, String actualContents, ITestOutputHelper outputHelper, Boolean warnOnDiffs, String baselineSubDir) in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs:line 49](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FBaselineHelper.cs&version=GBmain&_a=contents&line=49&lineEnd=50&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
[at Microsoft.DotNet.SourceBuild.SmokeTests.PoisonTests.VerifyUsage() in /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/PoisonTests.cs:line 30](https://dev.azure.com/dnceng/internal/_git/ddda0dd6-918f-48ba-ba8f-730d9d8c9320?path=%2Fvmr%2Ftest%2FMicrosoft.DotNet.SourceBuild.SmokeTests%2FPoisonTests.cs&version=GBmain&_a=contents&line=30&lineEnd=31&lineStartColumn=1&lineEndColumn=1&lineStyle=plain)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Furthermore the attached console output only includes the following:

Executing: git diff --no-index /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net9.0/assets/baselines/PoisonUsage.txt /vmr/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net9.0/logs/UpdatedPoisonUsage.txt

It requires intimate knowledge of the pipeline to know that the new UpdatedPoisonUsage.txt file is captured as a build artifact.

One way to improve the UX would be to include the diff in the standard output.

MichaelSimons commented 8 months ago

This is related in a way to https://github.com/dotnet/source-build/issues/3921.

ellahathaway commented 4 months ago

I can see three ways to resolve this:

1) Display a warning that there were content diffs & use the CreateBaselineUpdatePR tool 2) Display a warning that there were content diffs & give explicit output on where to find the file 3) Display a warning that there were content diffs & output the content diff