exercism / elm-test-runner

GNU Affero General Public License v3.0
3 stars 5 forks source link

Add parsing for `fuzz` tests family #49

Closed jiegillet closed 1 year ago

jiegillet commented 1 year ago

This is the sister PR to this one, where I'm trying to to use fuzz for the first time.

This PR makes it possible for the test runner to parse fuzz, fuzz2, fuzz3 and fuzzWith. For example, given the test

            , skip <|
                fuzzWith
                    { runs = 1000
                    , distribution =
                        Test.expectDistribution
                            [ ( Test.Distribution.atLeast 40, "has low charisma", \char -> char.charisma <= 12 )
                            , ( Test.Distribution.atLeast 40, "has high charisma", \char -> char.charisma > 12 )
                            , ( Test.Distribution.atLeast 40, "has low strength", \char -> char.strength <= 12 )
                            , ( Test.Distribution.atLeast 40, "has high strength", \char -> char.strength > 12 )
                            ]
                    }
                    (Fuzz.fromGenerator DndCharacter.character)
                    "generated characters are not all equal"
                <|
                    \_ -> Expect.pass

Will be parsed and shown to the students as

fuzzWith {runs = 1000
, distribution = Test.expectDistribution [(Test.Distribution.atLeast 40, "has low charisma", \char -> char.charisma <= 12)
, (Test.Distribution.atLeast 40, "has high charisma", \char -> char.charisma > 12)
, (Test.Distribution.atLeast 40, "has low strength", \char -> char.strength <= 12)
, (Test.Distribution.atLeast 40, "has high strength", \char -> char.strength > 12)]}
 (Fuzz.fromGenerator DndCharacter.character)
 "generated characters are not all equal"
 (\_ -> Expect.pass)

Note that for normal tests we only show the inside of the lambda, but for fuzz tests we need to show the full thing, because the arguments are required to understand the test (otherwise this example would only render Expect.pass).

For a wrong solution, the error message the student will see is

    Distribution of label "has high charisma" was insufficient:
      expected:  40.000%
      got:       0.000%.

    (Generated 1000 values.)

Note that the diagrams don't appear in this version of the messages.

jiegillet commented 1 year ago

At the moment this is very much set up to work with the DndCharacter exercise and tests, which is a particularly tricky situation, with the random / generators.

Do you think we will want to use fuzz testing more generally, and if so, do you think the messages will still be ergonomic in those cases? The messages for the DndCharacters tests are pretty complex. I'm a bit undecided as to whether this is the right thing or not, but maybe it is, as the situation is complex and it doesn't make sense to hide it, and it teaches students about what to expect when testing random / generators. However I think in more straightforward cases, the error messages could / should be clear and simple, even if we use fuzz tests.

I agree, the dnd-character exercise is tricky because I'm using fuzz tests on random generators, but in general I think we absolutely could use fuzz tests for other exercises. For example in wordy we have tests like

            test "multiplication" <|
                \() ->
                    Expect.equal (Just -75) <| answer "What is -3 multiplied by 25?"

We could absolutely do

        , fuzz2 Fuzz.int Fuzz.int "should be able to multiply arbitrary integers" <|
            \a b ->
                [ "What is ", String.fromInt a, " multiplied by ", String.fromInt b, "?" ]
                    |> String.concat
                    |> answer
                    |> Expect.equal (Just (a * b))

and in that case, if I used + rather than * in the implementation, I would get

↓ Wordy
✗ should be able to multiply arbitrary integers

Given (0,1)

    Just 1
    ╷
    │ Expect.equal
    ╵
    Just 0

We could also engineer better messages with fail if we wanted.