Running Test.cmd on a non english machine causes lots of unit tests to fail

[MaStr11]

Version Used:

Latest master at commit 69e99a8

Steps to Reproduce:

Run Test.cmd with a non english locale.

Expected Behavior:

All tests pass without errors.

Actual Behavior:

26 out of 41 test assemblies fail with errors. There were already about five failing tests in the past because of localization problems but since the introduction of xlf the situation got unbearable with hundreds of failing tests. I'm not sure but I think PR #23744 is to blame. Attached you can find the UnitTestResults.zip In the current situation I'm not able to run all tests locally to find regressions before committing. Maybe xUnit should be forced to use the en-US locale.

[tannergooding]

FYI. @jaredpar (compiler tests), @jinujoseph (IDE/Scripting tests)

[tannergooding]

Also FYI @jasonmalinowski

[MaStr11]

Update: Corrected the number of failures.

I did a grep 'Failures: <a href="#failures"><b>' *.* on my test results linked in the issue with this result (If I got this right, the number should show the number of failing tests):

Test	Failures	PR
Microsoft.CodeAnalysis.CSharp.Scripting.Desktop.UnitTests.dll.html	2	#24407
Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.dll.html	19	#24407
Microsoft.CodeAnalysis.VisualBasic.Scripting.UnitTests.dll.html	5	#24407
Roslyn.Compilers.CompilerServer.UnitTests.dll.html	21	#24460
Roslyn.Compilers.CSharp.CommandLine.UnitTests.dll.html	3	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.1.html	2	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.2.html	6	#24460
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.3.html	3	#24460
Roslyn.Compilers.CSharp.Semantic.UnitTests.dll.html	4	#24539
Roslyn.Compilers.CSharp.Symbol.UnitTests.dll.html	18	#24460
Roslyn.Compilers.CSharp.Syntax.UnitTests.dll.html	2	#24539
Roslyn.Compilers.VisualBasic.CommandLine.UnitTests.dll.html	1	#24460
Roslyn.Compilers.VisualBasic.Emit.UnitTests.dll.html	1	#24539
Roslyn.Compilers.VisualBasic.Semantic.UnitTests.dll.html	8	#24539
Roslyn.Compilers.VisualBasic.Symbol.UnitTests.dll.html	1	#24539
Roslyn.Compilers.VisualBasic.Syntax.UnitTests.dll.html	5	#24539
Roslyn.ExpressionEvaluator.CSharp.ExpressionCompiler.UnitTests.dll.html	7	#24424
Roslyn.ExpressionEvaluator.VisualBasic.ExpressionCompiler.UnitTests.dll.html	2	#24424
Roslyn.InteractiveHost.UnitTests.dll.html	35	#24407
Roslyn.Services.Editor2.UnitTests.dll.html	4	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.1.html	8	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.2.html	1	#24426
Roslyn.Services.Editor.CSharp.UnitTests.dll.5.html	4	#24426
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.1.html	2	#24426
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.3.html	2	#24426
Roslyn.Services.UnitTests.dll.html	6	#24426
Total	172

[jaredpar]

Hmm, I wonder if our non-English test passes of the unit tests have caught up with our new localization process. Seems like some sort of break down has happened here. Following up internally to see what's happening.

[jmarolf]

@dotnet/roslyn-infrastructure Can we have an Jenkins run that is non-english so we don't regress this once its fixed?

[jaredpar]

@jmarolf that is my intuition on how to fix this. But i'm not sure how well it's going to play with our XLF file process.

[tannergooding]

But i'm not sure how well it's going to play with our XLF file process.

Assuming we are pulling the expected value from the resource file and the actual value is being given by the product, it should just work. (CC. @tmeschter, @nguerrera)

[jasonmalinowski]

Agreed with @tannergooding. And yes, now that we have actual loc we can actually test said loc. 😄 Any idea @jaredpar if somebody already has a non-en-US image already setup in Jenkins we could piggy back off of?

[tmeschter]

@tannergooding Yes, I expect that most of these are issues where the test is hard-coding a value that it should instead be reading from the resources.

[iSeiryu]

I cloned the repo from scratch yesterday on an English locale machine and a lot of tests failed too.

git clone git@github.com:dotnet/roslyn.git
.\Restore.cmd
.\Build.cmd

Those steps went well with 0 failures.

Then I ran .\Test.cmd and after 38 minutes it gave me lots of reports named like /TestResults/Debug/WorkItem_8_x64_test_results.html with 70-99 Pass percentage. No report with 100%. I'm not sure how to get the right number of failed tests as I don't understand how to read those reports correctly.

Test execution time: 00:38:28.9071981
Test failures encountered
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll  --dotnet "C:\Program Files\dotnet\dotnet.exe"  --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug"  --configuration Debug  --tfm net472  --timeout 90  --include '\.UnitTests'  --exclude '\.InteractiveHost'  --html  --arch x64
System.Management.Automation.RuntimeException: Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll  --dotnet "C:\Program Files\dotnet\dotnet.exe"  --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug"  --configuration Debug  --tfm net472  --timeout 90  --include '\.UnitTests'  --exclude '\.InteractiveHost'  --html  --arch x64
at Exec-CommandCore, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 96
at Exec-Console, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 165
at TestUsingRunTests, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 468
at <ScriptBlock>, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 773
at <ScriptBlock>, <No file>: line 1

dotnet --info
.NET SDK:
 Version:           8.0.100
 Commit:            57efcf1350
 Workload version:  8.0.100-manifests.8d38d0cc

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.100\

Some error examples:

Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat​4ms
Error:
Assert.Equal() Failure
                        ↓ (pos 14)
Expected: PrintOptions {\n  Ellipsis: "...",\n  EscapeNonPrintableC···
Actual:   PrintOptions {\r\n  Ellipsis: "...",\r\n  EscapeNonPrintabl···
                        ↑ (pos 14)

Stack trace:
   at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat() in C:\Users\me\work\repos\github\roslyn\src\Scripting\CSharpTest\PrintOptionsTests.cs:line 72

Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.SourceGeneration.GeneratorDriverTests.Diagnostics_Respect_Pragma_Suppression​53ms
Error:
Expected:
    Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Actual:
    Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
Diff:
++>     Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
-->     Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Expected: True
Actual:   False

[iSeiryu]

I do have these settings in my git config --global

[core]
  autocrlf = input
  eol = lf

At least some of those failures relate to that because tests like these

var result = configTask.ConfigFileContents;
Assert.Equal(@"is_global = true
[c:/file1.cs]
build_metadata.Compile.ToRetrieve = abc123
build_metadata.AdditionalFile.ToRetrieve = def456
", result);

result in LF endings in all repo files and CRLF in all files outside of the repo.

We switched all of our work repos to LF even on Windows. There is no reason to keep them as CRLF. Maybe these test files like file1.cs can be created in a temp folder within the roslyn repo itself? The folder can be added to .gitignore or removed after the tests ran.

[sharwell]

@iSeiryu That would be an unrelated issue. I'm going to collapse the comments since they are off-topic for this issue, but please feel free to create a new issue with the content.