haf / expecto

A smooth testing lib for F#. APIs made for humans! Strong testing methodologies for everyone!
Apache License 2.0
663 stars 96 forks source link

Test.filter in the readme.md is wrong and it isn't clear how it should be adjusted #407

Closed abelbraaksma closed 3 years ago

abelbraaksma commented 3 years ago

The readme.md has the following example, which doesn't compile:

open Expecto
open MyLib.Tests
integrationTests // from MyLib.Tests
|> Test.filter (fun s -> s.EndsWith "another test") // the filtering function
|> runTestsWithCLIArgs [] [||]

However, the signature of Test.filter is joiner: string -> pred: (string list -> bool) -> Test -> Test. The description says "Filter tests by name". However, since the second argument takes a string list and returns a bool and not a (filtered) new list, it is unclear how this is supposed to work.

Same applies to the joiner argument. What does it do?

The readme suggests a signature like pred: (string -> bool) -> Test -> Test which seems more like the proper syntax for such a function.

How should the current function be used, what do the args mean, and is there a way to actually filter tests as explained in the readme.md? Likewise, the arguments to runTestsWithCLIArgs aren't explained. It stands to reason that they can deal with certain commandline arguments out of the box, perhaps a filter syntax like dotnet test has. Perhaps I missed this, but are these arguments explained somewhere?

PhilT commented 3 years ago

Possibly too late but here is what I do:

open Expecto
open System.Reflection

let testsFromModules =
  let assembly = Assembly.GetExecutingAssembly()

  assembly.GetTypes()
  |> Seq.filter (fun t -> t.Name <> "Main") // Ignore current file
  |> Seq.map (fun t ->
    t.GetMethods()
    |> Array.tryFind(fun m -> m.Name = "get_tests")
  )
  |> Seq.choose id
  |> Seq.map (fun func -> ((box (func.Invoke(null, [||]))) :?> Test))
  |> Seq.toList

let tests =
  testList "Matter" testsFromModules
  |> Test.shuffle "."

[<EntryPoint>]
let main argv =

  let config = {
    defaultConfig with
      colour = Logging.Colour256
  }

  runTestsWithArgs config argv tests

I use it to always shuffle tests before running them. Hope that helps.

abelbraaksma commented 3 years ago

@PhilT, tx for the example, though I'm a little surprised at the needed reflection. Ideally, the Readme should be updated with compilable code using lib functions, it's confusing to have incorrect examples there. Similar for the docs on the mentioned functions.