Is there a way to run all tests with shuffle?

[PhilT]

It seems you can either choose to use self-discovering tests with <[Tests]> and runTestsInAssemblyWithCLIArgs or be able to shuffle them but have to specify them explicitly.

myTests // from MyLib.Tests
|> Test.shuffle
|> runTestsWithCLIArgs [] [||]

Is there a way to shuffle auto-discovered tests?

[haf]

You could shuffle them where you assign them to a global field, but no, right now there's no way to do that.

[ZaymonFC]

This example may help although I am not entirely sure.

The project uses this method to hook into the standard dotnet test discovery.

[PhilT]

Thanks @ZaymonFC, that is close but it means specifying every module. I'm not familiar with .NET but I imagine there is a way to use reflection to grab all the modules in the test project and use that method to shuffle them. I'll investigate.

[PhilT]

That was easier than I was expecting!

open Expecto
open System.Reflection

let assembly = Assembly.GetExecutingAssembly()

let testsFromModules =
  assembly.GetTypes()
  |> Seq.filter (fun t -> t.ToString().StartsWith("ToplevelModuleName") && t.ToString().EndsWith("Test"))
  |> Seq.map (fun t -> (box (t.GetMethod("get_tests").Invoke(null, [||]))) :?> Test)
  |> Seq.toList

[<Tests>]
let tests =
  testList "Engine" testsFromModules
  |> Test.shuffle

[<EntryPoint>]
let main argv =

    printfn "Running tests!"

    let config =
        { defaultConfig with
            verbosity = Logging.LogLevel.Verbose
            colour = Logging.Colour256
            fsCheckMaxTests = 100
        }

    runTestsWithArgs config argv tests

I just need to change the StartsWith filter to be a bit more generic and then we can think about possibly adding this in to Expecto?

Thanks again @ZaymonFC your suggestion pushed me in the right direction 👍

I guess we could filter on any modules that have a get_tests method.

[PhilT]

This is a bit more generic:

open Expecto
open System.Reflection

let testsFromModules =
  let assembly = Assembly.GetExecutingAssembly()

  assembly.GetTypes()
  |> Seq.filter (fun t -> t.Name <> "Program") // Ignore current file
  |> Seq.map (fun t ->
    t.GetMethods() |> Array.tryFind(fun m -> m.Name = "get_tests")
  )
  |> Seq.choose id
  |> Seq.map (fun func -> ((box (func.Invoke(null, [||]))) :?> Test))
  |> Seq.toList

let tests =
  testList "Engine" testsFromModules
  |> Test.shuffle

[<EntryPoint>]
let main argv =

    let config =
        { defaultConfig with
            verbosity = Logging.LogLevel.Verbose
            colour = Logging.Colour256
            fsCheckMaxTests = 100
        }

    runTestsWithArgs config argv tests

[haf]

Nice!

I'm closing this for now, as the feature as such is a bit unclear. I'd love to improve expecto generally, but it has to be "needed" to avoid adding extra features for the sake of it. A nice PR could convince me ;)