The test Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.CommandLineRunnerTests.ReferenceSearchPaths1 is flaky

[brettfo]

It passes in CI but has failed in two successive signed builds, 20180430.4 and 20180430.5.

[sharwell]

What's the failure?

[gafter]

I see it failing on my desktop:

------ Test started: Assembly: Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.dll ------

Test 'Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.CommandLineRunnerTests.ReferenceSearchPaths1' failed: 
Expected:
C1 { }
C2 { }
C3 { }
C4 { }
Actual:
C:\Users\ngafter\AppData\Local\Temp\RoslynTests\CommandLineRunnerTests.cs_601_9e4f5cfa-35d4-4c8f-8bc4-80c5baea3be7.csx(5,11): error CS0246: The type or namespace name 'C1' could not be found (are you missing a using directive or an assembly reference?)
Differences:
++> C:\Users\ngafter\AppData\Local\Temp\RoslynTests\CommandLineRunnerTests.cs_601_9e4f5cfa-35d4-4c8f-8bc4-80c5baea3be7.csx(5,11): error CS0246: The type or namespace name 'C1' could not be found (are you missing a using directive or an assembly reference?)
--> C1 { }
--> C2 { }
--> C3 { }
--> C4 { }

Expected: True
Actual:   False
    D:\Roslyn\src\Test\Utilities\Portable\Assert\AssertEx.cs(367,0): at Roslyn.Test.Utilities.AssertEx.AssertEqualToleratingWhitespaceDifferences(String expected, String actual, Boolean escapeQuotes, String expectedValueSourcePath, Int32 expectedValueSourceLine)
    D:\Roslyn\src\Scripting\CSharpTest\CommandLineRunnerTests.cs(627,0): at Microsoft.CodeAnalysis.CSharp.Scripting.UnitTests.CommandLineRunnerTests.ReferenceSearchPaths1()

0 passed, 1 failed, 0 skipped, took 2.52 seconds (xUnit.net 2.3.1 build 3858).