VS Code/.NET Test Explorer and "dotnet test -t" do not list the individual scenarios

[deyanp]

I am surprised that VS Code's .NET Test Explorer (which uses "dotnet test -t" command) does not display the individual scenarios in the treeview, but only the features.

This means I cannot run/debug individual scenarios in a feature, I have to always run/debug all of them. Imagine 13 scenarios in a feature, and I want to troubleshoot only the last one ..

Is this by design (impossible to list scenarios due to the xUnit wiring)?

P.S: We are trying to use TickSpec with xUnit ...

[bartelink]

TL;DR its not easy but is conceptually possible - the messy stuff would be all in the xUnit bindings though. xUnit's Theory-leg discovery is a lazy and dynamic thing. While runners can surface the cases in the UI after they have been run, the underlying APIs don't have a way to say "run this case in the theory" - the lowest addressable thing is a Test Method. The TickSpec discovery APIs does have the full scenario list available, so it has the necessary things exposed. xUnit has a lower level Test Discoverer layer, but the last time I tried going that road (maybe 2y ago) I recall things getting incredibly messy - i.e. you end up rebuilding a lot of stuff as the default layer wants to discover and invoke via reflection (but only after you write a lot of serialization boilerplate).

[deyanp]

Thanks for the quick and detailed answer!

I assume there is no quick workaround ... I was thinking of creating 1 scenario per feature file, but that would significantly complicate the organization of the feature files ..

Coming from SpecFlow and big BDD Test suites of more than 10k scenarios organized in features, so a bit surprised to see that Tickspec has taken a different approach with the wiring ..

[bartelink]

The point of my first answer is to point out is that between the dotnet vstest infrastructure and xUnit legs of theories are not generally surfaced and/or addressible (and hence it's all about Test Methods.)

10k scenarios is mind blowing to me. In general, TickSpec is a minimalist implementation - its users and maintainers have not pushed for such enterprisy levels ;)

Whatever you do, you obviously want to retain the actual point of arranging stuff as Gherkin to suit the humans.

e.g. thinking aloud... if you're prepared to do/maintain some boilerplate in the F#, the sort of thing one can do relatively easy programatically with TickSpec as one exposes things to xUnit might be to codegen some wrapper methods so they are addressible as tests and then have a catch-all test that runs the rest - i.e. something like:

let ts = [| "scenario 1"; "scenario2" |]
let scenarios = discover "Me.feature"
let ``Suite1 Scenario1`` = run scenarios ts.[0] 
let ``Suite1 Scenario2`` = run scenarios ts.[1]
let ``Suite1 misc`` = scenarios |> Seq.except ts |> Seq.map run
let ``_GENERATE`` = scenarios |> generateCodeFromList // TODO paste results in to replace the above

Where run is code you write using the TickSpec object model.

Paging @mchaloupka @michalkovy who likely will have far better ideas

[deyanp]

Hmm, why do I see in this old presentation from 2010 (https://skillsmatter.com/skillscasts/1760-tickspec-bdd-for-c-sharp-f-sharp) that the scenarios are displayed in the VS Test Explorer?

So is the problem that I am using XUnit instead of NUnit or something else?

[bartelink]

Hm, clearly its been too long since I've thought about this or touched things (I have not touched it for >2 years other than reviewing things) - lets hope others can fact check my above answer :(

If you run in VS, resharper etc. you will see the theory items after the run with xUnit 2.

In xUnit v2 and the new discovery stuff in VS 2015 and later the discovery process is separated from the running much more. Its possible that NUnit does still do it the old way, but I don't have that paged in. The xUnit v1 stuff is still in the repo, and so is the NUnit - there is a good chance that the NUnit stuff from NUnit v2 will work with NUnit v3 (IIRC I tested it with V3) - IIRC the TickSpec.NUnit project and package has sample code (and the repo should too).

[deyanp]

I am running in VS Code with the extension .Net Core Test Explorer (https://marketplace.visualstudio.com/items?itemName=formulahendry.dotnet-test-explorer), and using XUnit ..

I am not willing to use Visual Studio + NUnit/Resharper etc, I was just giving that old example in order to identify where exactly is the problem - is it with VS Code, is it with the .Net Core Test Explorer extension of it, or with XUnit, or TickSpec ..

[deyanp]

One more note - it seems that after successful run the .Net Core Test Explorer expands the feature to the scenarios, but without stating the scenario names, only the feature name is repeated the number of scenarios there are:

[bartelink]

I think surfacing the name is doable (I think one needs to make the item the enumerator returns have a correct ToString impl). If you want to use the dot net core stuff only (good plan), then xunit most directly maps to that from what I know.

[deyanp]

FYI - moved to Rider, and there after successful execution you see the scenario names, so I guess it is not a TickSpec issue:

[bartelink]

Great to hear that you got sorted. Aside: I see xunit has a pre-enumerate theories config-switch; I wonder whether that might affect things or not wrt listing in other contexts.

[deyanp]

Tried to add a xunit.runner.json with the preEnumerate flag, but did not change anything in Rider (did not show the scenario names before run, also not the option to execute a single MemberData only).

What about: https://stackoverflow.com/questions/30574322/memberdata-tests-show-up-as-one-test-instead-of-many?

[bartelink]

Interesting find; I'd have to guess to be honest - I've not thought about, analysed or debugged such a thing for quite some time. In general, I'd say the TickSpec Test Suite is decent in terms of letting you analyse such matters; it may be worth experimenting in that context.

[bartelink]

Wild guess, still no analysis of any kind from me: https://github.com/xunit/xunit/pull/2019

[mchaloupka]

@deyanp Were you able to resolve this issue with xUnit and dotnet test -t command?

[deyanp]

@mchaloupka , actually yes (but we are using Rider for the past year, so haven't tested with VS Code), my colleague Zbigniew Kopczyk implemented it and wanted to create a PR for TickSpec ..

This is the code:

namespace rec Framework.Testing.TickSpecXunitWiring

open System.Diagnostics
open System.Reflection
open System.Collections.Generic
open System.Collections.Concurrent
open Xunit.Abstractions

type ScenarioKey = {
    FeatureFile: string
    ScenarioName: string
    Parameters: string[]
    Tags: string[]
}
module ScenarioKey =
    let empty() = {FeatureFile=""; ScenarioName=""; Parameters=[|""|]; Tags=[|""|]}

    let ofScenario (featureFile: string) (scenario:TickSpec.Scenario) =
        let parameters = scenario.Parameters |> Array.map (fun (k, v) -> sprintf "%s=%s" k v)
        {FeatureFile = featureFile; ScenarioName = scenario.Name; Parameters = parameters; Tags = scenario.Tags }

    let toSerializationInfo (info:IXunitSerializationInfo) (key:ScenarioKey) =
        info.AddValue("FeatureFile", key.FeatureFile)
        info.AddValue("ScenarioName", key.ScenarioName)
        info.AddValue("Parameters", key.Parameters)       
        info.AddValue("Tags", key.Tags)       

    let ofSerializationInfo (info:IXunitSerializationInfo) =
        let featureFile = info.GetValue<string>("FeatureFile")
        let scenarioName = info.GetValue<string>("ScenarioName")
        let parameters = info.GetValue<string[]>("Parameters")
        let tags = info.GetValue<string[]>("Tags")
        { FeatureFile = featureFile; ScenarioName = scenarioName; Parameters = parameters; Tags = tags}

    let toString (key:ScenarioKey) =
        if key.Parameters.Length = 0 && key.Tags.Length = 0 then key.ScenarioName
        else
            let parameters =  key.Parameters |> String.concat ","
            let tags =  key.Tags |> String.concat ","
            sprintf "%s<%s>{%s}" key.ScenarioName tags parameters

type IAssemblyStepDefinitionSource =
    abstract LoadScenariosFromEmbeddedResource: string -> TickSpec.Scenario seq

type ScenarioRepository() =
    static let mutable source: IAssemblyStepDefinitionSource option = None;
    static let sourceLock = obj
    static let mutable features: ConcurrentDictionary<string, ConcurrentDictionary<ScenarioKey, TickSpec.Scenario>> = ConcurrentDictionary();

    static let findSource () =
        //here be dragons!
        let frames = StackTrace().GetFrames()
        let methodFromMemberData =
            frames
            |> Seq.map (fun x -> x.GetMethod())
            |> Seq.map (fun x -> x, x.GetCustomAttributes())
            |> Seq.filter (fun (_, attributes) -> 
                if isNull attributes then
                    false
                else
                    attributes |> Seq.exists (fun x -> x :? Xunit.MemberDataAttribute)
                )
            |> Seq.map (fun (method, _) -> method)
            |> Seq.tryExactlyOne

        //assumption that every caller of scenariosfromembeddedresource will be in the correct assembly
        let index =
            frames
            |> Seq.mapi (fun i x -> i, x.GetMethod()) 
            |> Seq.filter(fun (i, x) -> x.Name = "ScenariosFromEmbeddedResource")
            |> Seq.map (fun (i,_) -> i)
            |> Seq.tryExactlyOne

        let methodFromExistingSource =
            index
            |> Option.map (fun x -> frames.[x + 1].GetMethod())

        let assembly = 
            match methodFromMemberData, methodFromExistingSource with
            | Some data, _ -> data.ReflectedType.Assembly
            | _, Some src -> src.ReflectedType.Assembly
            | None, None -> failwith "can not find assembly for stepdefinition source"

        AssemblyStepDefinitionsSource(assembly) :> IAssemblyStepDefinitionSource

    static member internal RegisterSource(newSource: IAssemblyStepDefinitionSource) =
        lock sourceLock (fun () -> 
            match source with
            | None -> source <- Some newSource
            | Some _ -> failwith "source has been already registered"
        )

    static member internal GetSource() =
         lock sourceLock (fun () ->
                match source with
                | None ->
                    let foundSource = findSource()
                    ScenarioRepository.RegisterSource (foundSource)
                    foundSource
                | Some source -> source
            )

    static member private LoadFeatureFile (featureFile:string) =
        let source = ScenarioRepository.GetSource()

        let loadScenarios featureFile = 
            let scenarios = source.LoadScenariosFromEmbeddedResource(featureFile)
            let dictionary =
                scenarios
                |> Seq.map (fun x -> KeyValuePair ((ScenarioKey.ofScenario featureFile x), x))
                |> ConcurrentDictionary
            dictionary

        features.GetOrAdd(featureFile, loadScenarios) |> ignore

    static member public LoadScenarios(featureFile) : TickSpec.Scenario seq =
        ScenarioRepository.LoadFeatureFile(featureFile)
        features.[featureFile].Values :> IEnumerable<TickSpec.Scenario>

    static member Action (key:ScenarioKey) =
        let featureFile = key.FeatureFile
        ScenarioRepository.LoadFeatureFile(featureFile)
        let scenarios = features.[featureFile]
        let scenario = scenarios.[key]
        scenario.Action

type Scenario(key:ScenarioKey) =
    let mutable key = key

    interface IXunitSerializable with
        member this.Serialize (info:IXunitSerializationInfo) =
            key |> ScenarioKey.toSerializationInfo info

        member this.Deserialize(info:IXunitSerializationInfo) =
            key <- info |> ScenarioKey.ofSerializationInfo

    new () = Scenario(ScenarioKey.empty())

    member this.Name
        with get() = key.ScenarioName

    member this.Action =
        ScenarioRepository.Action(key)

    member this.Tags
        with get() = key.Tags

    override this.ToString() =
        ScenarioKey.toString key

/// Represents a set of Step Definitions available within a given Assembly
type AssemblyStepDefinitionsSource(assembly : System.Reflection.Assembly) =
    let definitions = TickSpec.StepDefinitions(assembly)

    interface IAssemblyStepDefinitionSource with
        /// Yields Scenarios generated by parsing the supplied Resource Name and binding the steps to their Step Definitions
        member __.LoadScenariosFromEmbeddedResource resourceName : TickSpec.Scenario seq =
            let stream = assembly.GetManifestResourceStream(resourceName)
            definitions.GenerateScenarios(resourceName, stream)

    member this.ScenariosFromEmbeddedResource resourceName : Scenario seq =
        ScenarioRepository.LoadScenarios(resourceName)
        |> Seq.map (fun x -> ScenarioKey.ofScenario resourceName x)
        |> Seq.map Scenario

/// Adapts a Scenario Sequence to match the required format for an xUnit MemberData attribute
module MemberData =
    let ofScenarios xs = xs  |> Seq.map (fun x -> [| x |])

and then in your test project you use it like this:

namespace XxxService.Yyy.IntegrationTests
open Framework.Testing.TickSpecXunitWiring

static let source = AssemblyStepDefinitionsSource(System.Reflection.Assembly.GetExecutingAssembly())
static let scenarios resourceName = source.ScenariosFromEmbeddedResource resourceName |> MemberData.ofScenarios

[<Theory; MemberData("scenarios", "XxxService.Yyy.IntegrationTests.Features.Zzz.feature")>]
    member this.Zzz (scenario : Scenario) = 
        scenario.Action.Invoke()

After first run scenarios are displayed in the Unit Tests window in Rider, and can be debugged/run individually ..

[mchaloupka]

Awesome, thanks for responding. TickSpec was historically unopinionated regarding specific wiring. However, I can see that the wiring is getting longer so it would make sense to make it easier to use. I can imagine that there could be a specific nuget to handle the wiring. For example TickSpec.xUnit.

I would be happy to help and review the PR to introduce it. We have a gitter channel - it may be easier to discuss there the details. If not, feel free to just open a PR there and we can iterate there.

[mchaloupka]

The xUnit integration is released as separate nuget starting from release 2.0.2.