Visual Studio test discovery fails when FsCheckConfig.arbitrary set

[jackfoxy]

Reproducible in the "testadapter" branch of https://github.com/jackfoxy/personalServer

Visual Studio test discovery results in message (See output window, Tests):

'vstest.discoveryengine.x86.exe' WCF communication channel unavailable. Active request will be aborted.

Cause: appears to be problem with passing types in the FsCheckConfig.arbitrary field

Reproduction steps:

File: PersonalServer\tests\PersonalServer.Tests\DomainTypes.fs

At top of file is FsCheckConfig binding both with and without setting arbitrary list. Setting the list causes discovery to fail.

[misterspeedy]

I started getting this message after installing the adapter. It affected all solutions, not just the one where I installed the adapter (using paket). The problem persists even through multiple reboots. The install seems to be changing something global. (Was using VS2017 community with F# nightly on Win10.)

[misterspeedy]

Even uninstalling and re-installing VS2017 didn't fix this.

[misterspeedy]

One more bit of info - it affects solutions which use a custom FSCheck Generator. (Even those which didn't have the adapter installed and haven't been changed in any way before it was installed in another project). But it doesn't affect test discovery in solutions which don't use a generator.

[misterspeedy]

I wanted to check that what I said was true - that this affects projects other than the one where the adapter was installed. So I've just created a brand new project, installed just FSCheck and the FSCheck adapter (but not the Expecto adapter), and added this code:

namespace ExpectoRepro

open Xunit
open FsCheck
open FsCheck.Xunit

module public Generation =

    type ShortString = ShortString of string
        with
            override this.ToString() =
                match this with ShortString s -> s

    type Generators =
        static member ShortString() =
            Arb.Default.String()
            |> Arb.filter (fun s -> s <> null)
            |> Arb.filter (fun s -> s.Length < 10)
            |> Arb.convert ShortString string

[<PropertiesAttribute(Arbitrary = [| typeof<Generation.Generators> |]) >]
module Tests =

    open Generation

    [<Property>]
    let ``A short string is short`` (ShortString s) =
        s.Length < 10

When I built with VS's test explorer open, I again got:

------ Discover test started ------ [xUnit.net 00:00:00.6119947] Discovering: ExpectoRepro 'vstest.discoveryengine.x86.exe' WCF communication channel unavailable. Active request will be aborted. ========== Discover test finished: 0 found (0:00:04.3804718) ==========

This style of project was working perfectly before I first installed the Expecto adapter. It seems that something in the adapter itself, in paket, or in Visual Studio, is making a permanent global change which breaks all projects which use FSCheck Generators. I've uninstalled and reinstalled VS2017 three times and rebooted several times.

[misterspeedy]

When I open the project from the previous comment in VS2015 (as opposed to 2017) I get the same error. So it seems that whatever got changed is at the machine level rather than the VS level.

[misterspeedy]

This works fine on the ExpectoRepro project:

vstest.console.exe "C:\Users\kitea\Documents\Visual Studio 2017\Projects\ExpectoRepro\ExpectoRepro\bin\Debug\ExpectoRepro.dll"

So there is nothing inherently wrong with the test DLL.

[jackfoxy]

To clarify, my issue, which only happens when I use FsCheckConfig.arbitrary, is with Visual Studio 2015.

[jackfoxy]

I just upgraded my project to FSharp.Core 4.4.1 Still using VS 2015 And the test adapter still works so long as I don't try to map custom generators to arbitrary (so I have to test with the custom generators using the more verbose Prop.forAll

Anyway it it's any help my pakek.lock file reports the following for Expecto components:

Expecto (4.2.1)
  Argu (>= 3.7)
  FSharp.Core (>= 4.1.12)
  Mono.Cecil (>= 0.9.6.4)
Expecto.BenchmarkDotNet (4.2.1)
  BenchmarkDotNet (>= 0.10.5)
  Expecto (>= 4.2.1)
  FSharp.Core (>= 4.1.12)
Expecto.FsCheck (4.2.1)
  Expecto (>= 4.2.1)
  FsCheck (>= 2.8)
  FSharp.Core (>= 4.1.12)
Expecto.VisualStudio.TestAdapter (4.0.0-alpha007) - version_in_path: true
  Expecto (>= 4.0 < 5.0)

[misterspeedy]

All my tests also run perfectly in NCrunch. It does seem to be very specific to the combination of having installed the adapter, and having a generator.

Doing a paket uninstall on the adapter didn't fix it incidentally.

[thinkbeforecoding]

I'm having the same problem... and I also try to find where it comes from...

@misterspeedy , when running on NCrunch, which runner are you using ?

I was wondering whether it could be something that is usually fixed by binding redirects... If the adapter is trying to load a library with a version mismatched that is resolved when running the expecto projet by a binding redirect, it could be failing to load ??

[jackfoxy]

I re-tested this with adapter 5.0.0.2 and FSharp.Core 4.2.3 and the same problem persists.

I cleaned-up my test branch "testadapter" at https://github.com/jackfoxy/PersonalServer/tree/testadapter if someone feels inspired to try debugging.

[Julien-Pires]

I've got the same problem but with a different error message: "The active test discovery was aborted. Reason: Process is terminated due to StackOverflowException.".

The problem occurs when trying to change the arbitrary value of FsCheckConfig.

This could be reproduced easily with Expecto example about property based testing where they override arbitrary of the config.

This bug prevents us from using FsCheck and Visual Studio Test Explorer at the same time.

[olivercoad]

I've made a pull request which fixes this issue. Anyone want to give it a test?

[jackfoxy]

@olivercoad I'm interested in testing. I need to check if my test case still results in the problem.