Open MaStr11 opened 6 years ago
FYI. @jaredpar (compiler tests), @jinujoseph (IDE/Scripting tests)
Also FYI @jasonmalinowski
Update: Corrected the number of failures.
I did a grep 'Failures: <a href="#failures"><b>' *.*
on my test results linked in the issue with this result (If I got this right, the number should show the number of failing tests):
Test | Failures | PR |
---|---|---|
Microsoft.CodeAnalysis.CSharp.Scripting.Desktop.UnitTests.dll.html | 2 | #24407 |
Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.dll.html | 19 | #24407 |
Microsoft.CodeAnalysis.VisualBasic.Scripting.UnitTests.dll.html | 5 | #24407 |
Roslyn.Compilers.CompilerServer.UnitTests.dll.html | 21 | #24460 |
Roslyn.Compilers.CSharp.CommandLine.UnitTests.dll.html | 3 | #24460 |
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.1.html | 2 | #24460 |
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.2.html | 6 | #24460 |
Roslyn.Compilers.CSharp.Emit.UnitTests.dll.3.html | 3 | #24460 |
Roslyn.Compilers.CSharp.Semantic.UnitTests.dll.html | 4 | #24539 |
Roslyn.Compilers.CSharp.Symbol.UnitTests.dll.html | 18 | #24460 |
Roslyn.Compilers.CSharp.Syntax.UnitTests.dll.html | 2 | #24539 |
Roslyn.Compilers.VisualBasic.CommandLine.UnitTests.dll.html | 1 | #24460 |
Roslyn.Compilers.VisualBasic.Emit.UnitTests.dll.html | 1 | #24539 |
Roslyn.Compilers.VisualBasic.Semantic.UnitTests.dll.html | 8 | #24539 |
Roslyn.Compilers.VisualBasic.Symbol.UnitTests.dll.html | 1 | #24539 |
Roslyn.Compilers.VisualBasic.Syntax.UnitTests.dll.html | 5 | #24539 |
Roslyn.ExpressionEvaluator.CSharp.ExpressionCompiler.UnitTests.dll.html | 7 | #24424 |
Roslyn.ExpressionEvaluator.VisualBasic.ExpressionCompiler.UnitTests.dll.html | 2 | #24424 |
Roslyn.InteractiveHost.UnitTests.dll.html | 35 | #24407 |
Roslyn.Services.Editor2.UnitTests.dll.html | 4 | #24426 |
Roslyn.Services.Editor.CSharp.UnitTests.dll.1.html | 8 | #24426 |
Roslyn.Services.Editor.CSharp.UnitTests.dll.2.html | 1 | #24426 |
Roslyn.Services.Editor.CSharp.UnitTests.dll.5.html | 4 | #24426 |
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.1.html | 2 | #24426 |
Roslyn.Services.Editor.VisualBasic.UnitTests.dll.3.html | 2 | #24426 |
Roslyn.Services.UnitTests.dll.html | 6 | #24426 |
Total | 172 |
Hmm, I wonder if our non-English test passes of the unit tests have caught up with our new localization process. Seems like some sort of break down has happened here. Following up internally to see what's happening.
@dotnet/roslyn-infrastructure Can we have an Jenkins run that is non-english so we don't regress this once its fixed?
@jmarolf that is my intuition on how to fix this. But i'm not sure how well it's going to play with our XLF file process.
But i'm not sure how well it's going to play with our XLF file process.
Assuming we are pulling the expected value from the resource file and the actual value is being given by the product, it should just work. (CC. @tmeschter, @nguerrera)
Agreed with @tannergooding. And yes, now that we have actual loc we can actually test said loc. 😄 Any idea @jaredpar if somebody already has a non-en-US image already setup in Jenkins we could piggy back off of?
@tannergooding Yes, I expect that most of these are issues where the test is hard-coding a value that it should instead be reading from the resources.
I cloned the repo from scratch yesterday on an English locale machine and a lot of tests failed too.
git clone git@github.com:dotnet/roslyn.git
.\Restore.cmd
.\Build.cmd
Those steps went well with 0 failures.
Then I ran .\Test.cmd
and after 38 minutes it gave me lots of reports named like /TestResults/Debug/WorkItem_8_x64_test_results.html
with 70-99 Pass percentage. No report with 100%. I'm not sure how to get the right number of failed tests as I don't understand how to read those reports correctly.
Test execution time: 00:38:28.9071981
Test failures encountered
Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll --dotnet "C:\Program Files\dotnet\dotnet.exe" --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug" --configuration Debug --tfm net472 --timeout 90 --include '\.UnitTests' --exclude '\.InteractiveHost' --html --arch x64
System.Management.Automation.RuntimeException: Command failed to execute with exit code 1: C:\Program Files\dotnet\dotnet.exe C:\Users\me\work\repos\github\roslyn\artifacts\bin\RunTests\Debug\net7.0\RunTests.dll --dotnet "C:\Program Files\dotnet\dotnet.exe" --logs "C:\Users\me\work\repos\github\roslyn\artifacts\log\Debug" --configuration Debug --tfm net472 --timeout 90 --include '\.UnitTests' --exclude '\.InteractiveHost' --html --arch x64
at Exec-CommandCore, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 96
at Exec-Console, C:\Users\me\work\repos\github\roslyn\eng\build-utils.ps1: line 165
at TestUsingRunTests, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 468
at <ScriptBlock>, C:\Users\me\work\repos\github\roslyn\eng\build.ps1: line 773
at <ScriptBlock>, <No file>: line 1
dotnet --info
.NET SDK:
Version: 8.0.100
Commit: 57efcf1350
Workload version: 8.0.100-manifests.8d38d0cc
Runtime Environment:
OS Name: Windows
OS Version: 10.0.22631
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\8.0.100\
Some error examples:
Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat​4ms
Error:
Assert.Equal() Failure
↓ (pos 14)
Expected: PrintOptions {\n Ellipsis: "...",\n EscapeNonPrintableC···
Actual: PrintOptions {\r\n Ellipsis: "...",\r\n EscapeNonPrintabl···
↑ (pos 14)
Stack trace:
at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.UnitTests.PrintOptionsTests.ValidMemberDisplayFormat() in C:\Users\me\work\repos\github\roslyn\src\Scripting\CSharpTest\PrintOptionsTests.cs:line 72
Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.SourceGeneration.GeneratorDriverTests.Diagnostics_Respect_Pragma_Suppression​53ms
Error:
Expected:
Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Actual:
Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
Diff:
++> Diagnostic("GEN001", "omm", isSuppressed: true).WithLocation(2, 4)
--> Diagnostic("GEN001", "com", isSuppressed: true).WithLocation(2, 3)
Expected: True
Actual: False
I do have these settings in my git config --global
[core]
autocrlf = input
eol = lf
At least some of those failures relate to that because tests like these
var result = configTask.ConfigFileContents;
Assert.Equal(@"is_global = true
[c:/file1.cs]
build_metadata.Compile.ToRetrieve = abc123
build_metadata.AdditionalFile.ToRetrieve = def456
", result);
result in LF
endings in all repo files and CRLF
in all files outside of the repo.
We switched all of our work repos to LF
even on Windows. There is no reason to keep them as CRLF
. Maybe these test files like file1.cs
can be created in a temp folder within the roslyn repo itself? The folder can be added to .gitignore
or removed after the tests ran.
@iSeiryu That would be an unrelated issue. I'm going to collapse the comments since they are off-topic for this issue, but please feel free to create a new issue with the content.
Version Used:
Latest master at commit 69e99a8
Steps to Reproduce:
Run
Test.cmd
with a non english locale.Expected Behavior:
All tests pass without errors.
Actual Behavior:
26 out of 41 test assemblies fail with errors. There were already about five failing tests in the past because of localization problems but since the introduction of xlf the situation got unbearable with hundreds of failing tests. I'm not sure but I think PR #23744 is to blame. Attached you can find the UnitTestResults.zip In the current situation I'm not able to run all tests locally to find regressions before committing. Maybe xUnit should be forced to use the en-US locale.