search

dotnet / roslyn

The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.
https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/
MIT License
18.95k stars 4.02k forks source link

DefaultDiagnosticUpdateSourceTests contains flaky tests #45877

Open 333fred opened 4 years ago

333fred commented 4 years ago

The following are known flaky tests within this class:

Details for TestDefaultDiagnosticProviderSemantic

This test has failed 15 times in the last 14 days. Failure message:

Expected:
CS0246
Actual:
CS0246,
CS0246
Differences:
    CS0246,
++> CS0246

Expected: True
Actual:   False

Some examples of failures: https://dev.azure.com/dnceng/public/_build/results?buildId=724700&view=ms.vss-test-web.build-test-results-tab https://dev.azure.com/dnceng/public/_build/results?buildId=721954&view=ms.vss-test-web.build-test-results-tab https://dev.azure.com/dnceng/public/_build/results?buildId=718388&view=ms.vss-test-web.build-test-results-tab

One possibly notable detail: failures seem to take ~2.2 seconds for the test to run, successes take under .5 seconds for the most part.

Details for TestDefaultDiagnosticProviderRemove

Builds

Build Pull Request Test Failure Count
#739651 #46176 1
#739718 #46179 1
#739874 #46183 1
#739896 #45800 1
#740061 #46194 1
#740083 #46138 1
#740155 #45806 1
#740370 Rolling 1
#740378 Rolling 1
#740397 Rolling 1
#740404 #46189 1
#740484 #44447 1
#740543 #46199 1
#740831 Rolling 1
#741237 Rolling 1
#741316 Rolling 1
#741604 #46219 1
#741683 #46038 1

Configurations

Details for TestDefaultDiagnosticProviderAll

https://github.com/dotnet/roslyn/blob/1386d0fdd719f0a141d46315582f5b2724719299/src/VisualStudio/Core/Test/Diagnostics/DefaultDiagnosticUpdateSourceTests.vb#L143

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=749582&view=ms.vss-test-web.build-test-results-tab&runId=23240534&resultId=202015&paneView=debug

Xunit log:

Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.TestDefaultDiagnosticProviderAll [STARTING]

      Expected:
      CS1002
      CS0246
      Actual:
      CS1002
      CS1002
      CS0246
      CS0246
      Differences:
          CS1002
      ++> CS1002
          CS0246
      ++> CS0246

      Expected: True
      Actual:   False
      Stack Trace:
        /_/src/Test/Utilities/Portable/Assert/AssertEx.cs(442,0): at Roslyn.Test.Utilities.AssertEx.SetEqual[T](IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector)
        /_/src/Test/Utilities/Portable/Assert/AssertEx.cs(425,0): at Roslyn.Test.Utilities.AssertEx.SetEqual(IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector)
        /_/src/VisualStudio/Core/Test/Diagnostics/DefaultDiagnosticUpdateSourceTests.vb(177,0): at Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.VB$StateMachine_4_TestDefaultDiagnosticProviderAll.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    Microsoft.VisualStudio.LanguageServices.UnitTests.Diagnostics.DefaultDiagnosticUpdateSourceTests.TestDefaultDiagnosticProviderAll [FINISHED] Time: 1.6519875s
mavasani commented 4 years ago

I'll take a look.

mavasani commented 4 years ago

Assigning to @sharwell based on the investigations in https://github.com/dotnet/roslyn/issues/46255

sharwell commented 4 years ago

@mavasani This doesn't appear to be the same issue as #46255. ☹️

RikkiGibson commented 4 years ago

Another failure in https://dev.azure.com/dnceng/public/_build/results?buildId=752137&view=logs&j=25fa373a-98a5-5235-2ed8-ead0b7a28f80&t=2f2a122e-a512-5643-f332-5c2c88cb7bf1&s=d654deb9-056d-50a2-1717-90c08683d50a

sharwell commented 3 years ago

This seems to affect this line: https://github.com/dotnet/roslyn/blob/075e9c3e3e57b150cfae938d4b652b9ee9c4171f/src/Compilers/CSharp/Portable/Binder/Binder_Symbols.cs#L2386

It feels like parallel binding of the same identifier (optimistic initialization) is resulting in a case where one of the side effects (error reporting) is observed from both the path that wins the initialization and the one that doesn't.