Flaky TransparentCompiler fuzzing test

[majocha]

Recently reenabled fuzzing test still needs some work: https://github.com/dotnet/fsharp/blob/373fb23c5a6a7a00ddee9d142150cb18b2bd51d5/tests/FSharp.Compiler.ComponentTests/FSharpChecker/TransparentCompiler.fs#L603-L607

It is sporadically failing in CI.

It takes a lot of iterations locally to trip it, but it is possible to repro.

A good way to get a repro locally, (and to stress test a selected test case in general) is to define a xUnit attrbute like this

type RepeatAttribute(count, [<ParamArray>] args: obj array) =
    inherit DataAttribute()
    override _.GetData _ = Seq.init count (fun i -> [| yield! args; i |])

and apply it like this:

[<Theory>]
[<Repeat(100, SignatureFiles.Yes)>]
[<Repeat(100, SignatureFiles.No)>]
[<Repeat(100, SignatureFiles.Some)>]
let Fuzzing signatureFiles _ =

Running the theory in parallel it can be repro'd in around 100 iterations. Out of curiosity I tried to run this test with useTransparentCompiler = false and it also fails from time to time. It is possible that it reveals some concurrency issue unrelated to TransparentCompiler.

[0101]

Let's skip it for now then.

Out of curiosity I tried to run this test with useTransparentCompiler = false and it also fails from time to time. It is possible that it reveals some concurrency issue unrelated to TransparentCompiler.

That's quite interesting, does it fail with the same error?

[majocha]

That's quite interesting, does it fail with the same error?

I thought so, but I'm not so sure now as I can't get that result anymore.

[majocha]

Yes, occasional Expected no errors, but there were some: when useTransparentCompiler = false in fuzzingTest

Assert.Equal() Failure: Collections differ
Expected: []
Actual:   [Tuple (AbortedChecking
  "Expected no errors, but there were some: 
[|E:\repos\fsharp\artifacts\bin\FSharp.Compiler.ComponentTests\Release\net9.0\test-projects\TestProject_680b94c6\FileF012.fs (3,5)-(3,6) typecheck error Module 'TestProject_680b94c6.ModuleF012' contains
    val f: x: 'a -> (ModuleF001.TF001V_1<'a> * 'a0 * ModuleF004.TF004V_1<'a>) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * ModuleF006.TF006V_1<'a>) * (ModuleF001.TF001V_1<'a> * ((ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * 'b) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * 'c) * ModuleF004.TF004V_1<'a>) * ModuleF007.TF007V_1<'a>) * ModuleF010.TF010V_1<'a>) * TF012V_1<'a>    
but its signature specifies
    val f: x: 'a -> (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * 'a0) * ModuleF004.TF004V_1<'a>) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * ModuleF006.TF006V_1<'a>) * (ModuleF001.TF001V_1<'a> * ((ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * 'b) * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * (ModuleF001.TF001V_1<'a> * ModuleF002.TF002V_1<'a>) * 'c) * ModuleF004.TF004V_1<'a>) * ModuleF007.TF007V_1<'a>) * ModuleF010.TF010V_1<'a>) * TF012V_1<'a>    
The types differ|]", 1)