Feature request: Add a way to assert that a test fails

[jfmengels]

Context

In elm-review I provide a testing module that lets you run a lot of custom assertions so you don't have to write them themselves.

The problem with this testing module is that I can't test it. I can't write a test that says "in this scenario, I expect the testing module to cause a test failure that fails with this error message"

At the moment, I have to do manual testing to make sure that the test fails as expected (meaning I don't have a net to catch me if I ever mess it up). I also write separate tests to make sure that the error messages look as expected, but I'm basically testing the error message function, not the assertion itself (https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm).

Proposal

I would like to propose an Expect.toFail function.

{-| Expect the test to fail with the given message.

This is useful if you wish to ensure that custom test assertions work as expected.
-}
toFail : (String -> Expectation) -> Expectation -> Expectation

Usage

test "..." <|
    \() ->
        input
            |> someTestUtility
            -- Have this test pass if the test "so far" fails
            |> Expect.toFail (\_ -> Expect.pass)

test "..." <|
    \() ->
        input
            |> someTestUtility
            -- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
            -- and have it fail otherwise.
            |> Expect.toFail (\error -> String.contains "incorrect location" test |> Expect.equal True)

I think it would be possible to have the function compare the resulting error and match it against the expected error string, but I feel that this is more general and powerful.

In my use-case, I will likely compare the error with some exact string, but I will also make sure that all lines in the error message are less than 76 characters long, otherwise the output is really ugly when presented to the user. I will likely do something like this:

test "..." <|
    \() ->
        input
            |> someTestUtility
            -- Have this test pass if the test "so far" fails with a message that contains "incorrect location"
            -- and have it fail otherwise.
            |> Expect.toFail (expectMessageEqual "....")

-- Directly taken from
-- https://github.com/jfmengels/elm-review/blob/master/tests/Review/Test/FailureMessageTest.elm#L45-L59
expectMessageEqual : String -> String -> Expectation
expectMessageEqual expectedMessage =
    Expect.all
        [ Expect.equal <| String.trim expectedMessage
        , \receivedMessage ->
            Expect.all
                (String.lines receivedMessage
                    |> List.map
                        (\line () ->
                            (String.length line <= 76)
                                |> Expect.true ("Message has line longer than 76 characters:\n\n" ++ line)
                        )
                )
                ()
        ]

Alternatively, the first argument could be (String -> Bool) but it could be nice to have the same API, and also to have nice error messages. If we do go with (String -> Expectation), there is a need to figure out what the error message would look like when wrapped inside a Expect.toFail.

Afterword

Having this available would help me be a lot more confident about this testing module that I provide to my users. Currently, this is quite a lot of work to separate the error message from the test failures, and it is still somewhat unreliable and requires manual testing.

I don't know if it would make sense to have a corresponding Expect.toSucceed but I don't believe I have a need for it.

[Janiczek]

Would it be possible to use Test.Runner.getFailureReason and check something about the Just (if failing) or Nothing (if succeeded) in there?

[jfmengels]

It sounds like it should yes 🤔

My attempt so far has been unfruitful because the received description and the expected one differ according to elm-test, even though as far as I (and a difftool) can tell they're the same.

[Janiczek]

@jfmengels Is the childre typo in there intentional? I feel like it is, but better check, in case that solves the issue :)

[avh4]

I have such an assertion working in elm-program-test https://github.com/avh4/elm-program-test/blob/main/tests/Test/Expect.elm#L20-L42= (might be a bit more than you want, since it also strips out ANSI escape sequences from the failure message, and it takes a list of lines instead of the exact string so it's easier for me to indent big multiline strings in the tests themselves.

[jfmengels]

Is the childre typo in there intentional?

Yes, I was trying to create a test that expects the test to fail, which I forced by expecting the wrong thing in the first part.

I may have done something wrong, but the error message I'm currently getting is something simplifiable to I got "abc" but expected "abc", which is not helpful. I'll take a closer look when I have a bit more time.

I have such an assertion working in elm-program-test

Thank, I'll check that out. I'm also putting ANSI escapes codes in my messages and I'd like to validate that they're there. (FYI, this is what I do in a separate part of elm-review: test what the thing looks like with and without colors https://github.com/jfmengels/node-elm-review/blob/master/template/tests/ReporterTest.elm#L125-L151)