fscheck / FsCheck

Random Testing for .NET
https://fscheck.github.io/FsCheck/
BSD 3-Clause "New" or "Revised" License
1.15k stars 154 forks source link

Check.QuickAll does not find testables that return unit #652

Closed isberg closed 6 months ago

isberg commented 6 months ago

According to the documentation the following is true:

FsCheck can check properties of various forms - these forms are called testable, and are indicated in the API by a generic type called 'Testable. A 'Testable may be a function of any number of parameters that returns bool or unit. In the latter case, a test passes if it does not throw.

It is the latter case that does not seem true when using F# and Check.QuickAll as in this code:

open FsCheck

type Tests =
    static member ``should be true`` : bool = true
    static member ``should not throw an exception`` : Unit = ()

Check.Quick Tests.``should be true``
Check.Quick Tests.``should not throw an exception``

Check.QuickAll<Tests> ()

For each of the call to Check.Quick the ouput is Ok, passed 100 tests. as expected, but the output from Check.QuickAll is only:

--- Checking Tests ---
Tests.get_should be true-Ok, passed 100 tests.

The latter case and the second test seems to be ignored by Check.QuickAll. :-( I have tried with both the current version and the 3.0.0-rc1, with the same result.

kurtschelfthout commented 6 months ago

A 'Testable may be a function of any number of parameters that returns bool or unit

Your members are not functions. Try static member ``should be true``() = true

isberg commented 6 months ago

Thanks @kurtschelfthout! That input solves my issue. I did some more digging though and found a few confusing things that I want to report for anyone else having similar problems. All the confusion relates to "functions" returning unit behaving different than functions returning bool:

open FsCheck

type Tests =
    static member ``case 0`` : Unit -> Unit = fun () -> () // :-)
    static member ``case 1`` = fun () -> () // :-)
    static member ``case 2`` () = () // :-(
    static member ``case 3`` (x:int) : Unit = () // :-(
    static member ``case 4`` : int -> Unit = fun x -> () // :-)
    static member ``case 5`` = fun x -> () // :-)
    static member ``case 6`` (x:int) = x = x // :-)
    static member ``case 7`` (x:int) : bool = x = x // :-)
    static member ``case 8`` x = x = x / 1 // :-)
    static member ``case 9`` : int -> bool = fun x -> x = x  // :-)

Check.QuickAll<Tests> ()

Check.Quick ("case 2", Tests.``case 2``)
Check.Quick ("case 3", Tests.``case 3``)

The output is:

--- Checking Tests ---
Tests.get_case 0-Ok, passed 100 tests.
Tests.get_case 1-Ok, passed 100 tests.
Tests.get_case 4-Ok, passed 100 tests.
Tests.get_case 5-Ok, passed 100 tests.
Tests.case 6-Ok, passed 100 tests.
Tests.case 7-Ok, passed 100 tests.
Tests.case 8-Ok, passed 100 tests.
Tests.get_case 9-Ok, passed 100 tests.

case 2-Ok, passed 100 tests.
case 3-Ok, passed 100 tests.

I also see that the output sometimes prepends get_ to the name of the test...

This leaves me in a situation where I just have to adapt the format of the test members, depending if they return bool or unit, which is fine, but potentially confusing when in the future someone tries to conform to a single format.