Running Test.cmd causes lots of unit tests to fail

[iSeiryu]

Version Used: Latest commit on main https://github.com/dotnet/roslyn/pull/71257 Steps to Reproduce:

1. git config --global -e
2. set these options

   [core]
   autocrlf = input
   eol = lf

3. Run

   git clone git@github.com:dotnet/roslyn.git
   .\Restore.cmd
   .\Build.cmd

4. Run .\Test.cmd. After 35+ minutes observe a lot of failed tests

Expected Behavior: All tests should pass. Actual Behavior: It gave me a lot of reports named like /TestResults/Debug/WorkItem_8_x64_test_results.html with 70-99 Pass percentage. No report with 100%.

Test execution time: 00:38:28.9071981
Test failures encountered
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll  --dotnet "C:\Program Files\dotnet\dotnet.exe"  --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug"  --configuration Debug  --tfm net472  --timeout 90  --include '\.UnitTests'  --exclude '\.InteractiveHost'  --html  --arch x64
System.Management.Automation.RuntimeException: Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll  --dotnet "C:\Program Files\dotnet\dotnet.exe"  --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug"  --configuration Debug  --tfm net472  --timeout 90  --include '\.UnitTests'  --exclude '\.InteractiveHost'  --html  --arch x64
at Exec-CommandCore, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 96
at Exec-Console, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 165
at TestUsingRunTests, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 468
at <ScriptBlock>, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 773
at <ScriptBlock>, <No file>: line 1

Some error examples:

Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat​4ms
Error:
Assert.Equal() Failure
                        ↓ (pos 14)
Expected: PrintOptions {\n  Ellipsis: "...",\n  EscapeNonPrintableC···
Actual:   PrintOptions {\r\n  Ellipsis: "...",\r\n  EscapeNonPrintabl···
                        ↑ (pos 14)

Stack trace:
   at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat() in C:\Users\me\work\repos\github\roslyn\src\Scripting\CSharpTest\PrintOptionsTests.cs:line 72

Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.SourceGeneration.GeneratorDriverTests.Diagnostics_Respect_Pragma_Suppression​53ms
Error:
Expected:
    Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Actual:
    Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
Diff:
++>     Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
-->     Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Expected: True
Actual:   False

At least some of those failures relate to CRLF vs LF because tests like these

var result = configTask.ConfigFileContents;
Assert.Equal(@"is_global = true
[c:/file1.cs]
build_metadata.Compile.ToRetrieve = abc123
build_metadata.AdditionalFile.ToRetrieve = def456
", result);

result in LF endings in all repo files and CRLF in all files outside of the repo.

We switched all of our work repos to LF even on Windows. There are some downsides and 0 benefits to keep them as CRLF. Maybe these test files like file1.cs can be created in a temp folder within the roslyn repo itself? The folder can be added to .gitignore or removed after the tests ran.

dotnet --info
.NET SDK:
 Version:           8.0.100
 Commit:            57efcf1350
 Workload version:  8.0.100-manifests.8d38d0cc

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.100\

[iSeiryu]

I tried resetting all LF files to CRLF via updating git config to use CRLF and then running these commands:

git rm -rf --cached .
git reset --hard HEAD

Some tests started passing when I run them via VS2022. But when I run .\Test.cmd -testCoreClr it still gives me a lot of errors.

Test execution time: 00:16:36.2049844
Test failures encountered
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll  --dotnet "C:\Program Files\dotnet\dotnet.exe"  --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug"  --configuration Debug  --tfm net6.0 --tfm net7.0 --tfm net8.0  --timeout 90  --tfm net6.0-windows  --include '\.UnitTests'  --html  --arch x64

[BrianMar]

I'm seeing the same kinds of issues with every pull I've done so far.

• Running on Windows with git defaults (core.autocrlf = true) gives me a small number of error cases, all of which appear to be CRLF based. For example:

  
   Source: SuppressionTests.vb line 907
   Duration: 16 ms
  
  Message: 
  Actual and expected values differ. Expected shown in baseline of diff:
  <LF>
  -Imports System<LF>
  -<LF>#Disable Warning ErrorDiagnostic ' ErrorDiagnostic<LF>
  -Class C<LF>
  -#Enable Warning ErrorDiagnostic ' ErrorDiagnostic<LF>
  +Imports System<CR><LF>
  +<CR><LF>
  +#Disable Warning ErrorDiagnostic ' ErrorDiagnostic<CR><LF>
  +Class C<CR><LF>
  +#Enable Warning ErrorDiagnostic ' ErrorDiagnostic<CR><LF>
   Sub Method()<LF>
   End Sub<LF>
  End Class

Expected: True Actual: False

When I've tried using the DevContainer from a Windows box, I'm getting a large number of failures regardless of whether I set core.autocrlf to "true" or "false".