search

dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
261 stars 127 forks source link

UX when there is failing SDK Diff and License Scan Tests could be improved. #4329

Open ellahathaway opened 2 months ago

ellahathaway commented 2 months ago

Related to https://github.com/dotnet/source-build/issues/3910 and https://github.com/dotnet/source-build/issues/4093

When running the SDK diff and license scan tests, extra knowledge is needed to know where to look for the updated files.

Additionally, when doing the SDK diff tests, it can sometimes be difficult to determine which test failed:

Determining projects to restore...
  All projects are up-to-date for restore.
  TestUtilities -> /mnt/vss/_work/1/s/artifacts/bin/TestUtilities/linux/Release/TestUtilities.dll
  Microsoft.DotNet.SourceBuild.SmokeTests -> /mnt/vss/_work/1/s/artifacts/bin/Microsoft.DotNet.SourceBuild.SmokeTests/linux/Release/Microsoft.DotNet.SourceBuild.SmokeTests.dll
/mnt/vss/_work/1/s/.dotnet/sdk/9.0.100-preview.4.24178.10/Microsoft.TestPlatform.targets(46,5): error : [xUnit.net 00:00:20.21]     Microsoft.DotNet.SourceBuild.SmokeTests.SdkContentTests.CompareMsftToSbFileList [FAIL] [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs(76): error VSTEST1: Microsoft.DotNet.SourceBuild.SmokeTests.SdkContentTests.CompareMsftToSbFileList: Microsoft.DotNet.SourceBuild.SmokeTests.BaselineHelper.CompareFiles(String expectedFilePath, String actualFilePath, ITestOutputHelper outputHelper, Boolean warnOnDiffs) Assert.Null() Failure: Value is not null [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs(76): error VSTEST1: Expected: null [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/BaselineHelper.cs(76): error VSTEST1: Actual:   "\nExpected file '/mnt/vss/_work/1/s/artifacts/bin/"··· [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
/mnt/vss/_work/1/s/.dotnet/sdk/9.0.100-preview.4.24178.10/Microsoft.TestPlatform.targets(46,5): error : [xUnit.net 00:00:24.66]     Microsoft.DotNet.SourceBuild.SmokeTests.ArtifactsSizeTest.CompareArtifactsToBaseline [FAIL] [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/ArtifactsSizeTest.cs(195): error VSTEST1: Microsoft.DotNet.SourceBuild.SmokeTests.ArtifactsSizeTest.CompareArtifactsToBaseline: Microsoft.DotNet.SourceBuild.SmokeTests.ArtifactsSizeTest.ReportDifferences() Differences were found in the artifacts sizes. [/mnt/vss/_work/1/s/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj]
  Passed Microsoft.DotNet.SourceBuild.SmokeTests.SdkContentTests.CompareMsftToSbAssemblyVersions [27s 109ms]
  Outputs:
    Standard Output Messages:
   Executing: tar xzf /mnt/vss/_work/1/Artifacts/dotnet-sdk-9.0.100-preview.2.24163.1-centos.9-x64.tar.gz -C /tmp/414eaz4n.b1m/sb
   Executing: tar xzf /mnt/vss/_work/1/Artifacts/dotnet-sdk-9.0.100-preview.2.24163.3-linux-x64.tar.gz -C /tmp/414eaz4n.b1m/msft
   Executing: git diff --no-index msft_assemblyversions.txt sb_assemblyversions.txt

  Failed! - Failed: 2, Passed: 1, Skipped: 0, Total: 3, Duration: [47s 824ms]

One way to improve the UX would be to include the diff in the standard output. Another way would be to clearly summarize which tests failed with clear instructions on how to view the updated files.

ellahathaway commented 1 month ago

With the inclusion of the CreateBaselineUpdatePR tool, I think all that really needs to happen is to output a better summary of the failing tests. Something like:

  Failed! - Failed: 2, Passed: 1, Skipped: 0, Total: 3, Duration: [47s 824ms]

- "##vso[task.logissue type=error;] ArtifactsSizeTest.CompareArtifactsToBaseline: Differences were found in the artifacts sizes"
- "##vso[task.logissue type=error;] SdkContentTests.CompareMsftToSbFileList: Content differences were found"
- "##vso[task.logissue type=debug;] View the updated file(s) in the created PR or in <path to artifacts>"