Reporting unconditional test failure, with data

[kkm000]

I am not sure that I am using Unquote in tests the way it was designed to be used, but I often find myself writing expressions that report a failure by abusing the operator =!. To give an actual example,

   [<Test>]
   let ``Alternatives are parsed with the lowest precedence``() =
      match strucAlt "a b | c d e" with
      | [ OneOf [ (_,l); (_,r) ] ] ->
         l =! strucSeq "a b"
         r =! strucSeq "c d e"
      | x -> [] =! [x]  // This always fails--an Unquote hack.

strucAlt and strucSeq are deconstructors, part of the test scaffolding. They are proven to perform correctly in their own tests, and invoke the actual function under test and transform results into a form that is easier to reason about. An evaluation backtrace from these would be quite verbose and not really interesting; reporting just the indigestible x in the | x -> branch is enough. The hack [] =! [x] does just that, but is quite ugly, wraps the printout into an extra list, and also requires that x : 'T when 'T : equality, which I'm got bitten by once.

Of course, I could write instead

test <@
      match strucAlt "a b | c d e" with
      | [ OneOf [ (_,l); (_,r) ] ] ->
         l = strucSeq "a b" &&
         r = strucSeq "c d e"
      | _ -> false
     @> 

but it seems to me this adds little to the readability of either the test itself or its output on failure. The interesting part will be buried in the middle of the trace.

Am I missing a function that would just fail the test unconditionally, like | x -> fail x? Is there an interest in adding one, if there is really none? It does not seem hard, reading the code of the operator =! and friends, as they already call an (inaccessible) failure function. I contemplated sending a PR, but then thought I should better ask first.

[stephen-swensen]

Hi @kkm000 - how about just throwing an exception, like

   [<Test>]
   let ``Alternatives are parsed with the lowest precedence``() =
      match strucAlt "a b | c d e" with
      | [ OneOf [ (_,l); (_,r) ] ] ->
         l =! strucSeq "a b"
         r =! strucSeq "c d e"
      | x -> 
        failwithf "Unexpected result, x=%A" x

[kkm000]

Thank you, I think I see what you mean. To make sure I understand your reply in relation to Unquote design, let me restate what I grok from it, in a more general form. True, false, or not even wrong: such an unconditional failure of Unquote.test (and related syntactic sweeteners like =! and friends) does not really belong with the Unquote design, and must be rather delegated to the testing framework's (e. g., NUnit in my case) facilities directly?

Or, yet in other words. I would be more than willing to contribute such a function, iff it is not going against the grain of Unquote. I just want to make sure that your statement isn't merely a suggested workaround, but is The Right Thing bona fide :)

[stephen-swensen]

@kkm000 - generally I think one of the things that has made unquote successful is that there are so few operators to learn. For example, assertions themselves mostly lean on the expressive power of F# itself (e.g. structural comparison). Where unquote just captures and conveys the result of expressions in a friendly way. So in this case, I think it is perfectly legitimate to delegate to the testing framework facilities (e.g. raising an exception or even using NUnit's Assert.Fail).

That said, I could be convinced about adding an unconditional failure operator to unquote if it added some real compelling value unique to what unquote is able to do. Note that only one new assertion operator has been added to unquote over the past 7 years!... the trap operator, which returns the value of the quoted expression but "traps" any (typically unexpected) exceptions and reports on them with expression decompilation. e.g.

let x = Some(3) 
let y = trap <@ x.Value @> // we "know" x.IsSome, but just in case we are wrong...
test <@ y = 3 @> // this is what we really wanted to test

[kkm000]

Totally convincing. Let me quickly check if the exception or NUnit.Assert does not cut it for some reason. That's unlikely, just want to be sure. I'll let you know in a day or two.

Yes, Unquote is simply amazing, thank you for it!

[kkm000]

Yes, Assert.Fail works just the same, so defining a test failure function in terms of it is a no-brainer.

  | x -> Assert.Fail("{0}", x)

How did not I think of this! Thanks. So I am closing the issue.