Assertion failure messages aren't what I hoped for

[robmoore-i]

Here's an example of the output of a failing test:

Raised at file "src/lib/ounit2/advanced/oUnitAssert.ml", line 45, characters 2-27
Called from file "src/lib/ounit2/advanced/oUnitRunner.ml", line 83, characters 13-26

not equal
------------------------------------------------------------------------------
Ran: 7 tests in: 1.16 seconds.
FAILED: Cases: 7 Tried: 7 Errors: 0 Failures: 1 Skip:  0 Todo: 0 Timeouts: 0.

It doesn't even show expected vs actual - it just says "not equal" which I find quite disappointing. This lack of information about the failure hugely reduces the usefulness of the tests, and inclines me to add my own print statements into the tests. In general, I'd like the test framework to provide descriptive failure messages for common sense assertions like 'assert_equal'.

I'm open to learning that most OCaml programmers don't care, but from the perspective of someone who has been embedded in TDD for a long time within other languages, I would quite like this feature, and I suspect it's quite a low-hanging fruit. I'm not raising this issue to throw rocks at OUnit (which I was so happy to come across!), but just to let the maintainers know that this is a feature that I and others new to the language would certainly appreciate.

Best wishes, stay well, and many thanks for creating this package, Rob

---

Edit note: Here is an example of the code I write to get the information I want from my test failures:

let test_last_seven_in_response_body _ =
  let actual = (Lwt_main.run (Oto.last_seven_in_response_body "www.amazon.co.uk")) in
  let expected = "</html5>" in
  try assert_equal expected actual
  with e ->
    Format.eprintf "\nError: %s\n%!" (Caml.Format.sprintf "Exn raised: %s" (Base.Exn.to_string e));
    print_string "Expected: ";
    print_endline expected;
    print_string "Actual:   ";
    print_endline actual;
    raise e;;

Which will output:

......
Error: Exn raised: ("OUnitTest.OUnit_failure(\"not equal\")")
Expected: </html5>
Actual:   </html>
F
==============================================================================
Error: OneToOneTest:6:test_last_seven_in_response_body.

File "/Users/romo/Documents/one-to-one/_build/oUnit-OneToOneTest-Robs-MacBook-Pro.local#02.log", line 8, characters 1-1:
Error: OneToOneTest:6:test_last_seven_in_response_body (in the log).

File "test/one_to_one_test.ml", line 35, characters 1-1:
Error: OneToOneTest:6:test_last_seven_in_response_body (in the code).

not equal
------------------------------------------------------------------------------
Ran: 7 tests in: 0.12 seconds.
FAILED: Cases: 7 Tried: 7 Errors: 0 Failures: 1 Skip:  0 Todo: 0 Timeouts: 0.

[gildor478]

Hi,

OCaml is not very flexible on the typing and it makes it complicated to implement an "assert_equal" that can print numbers, strings and other structures. This is a limitation of the language, everything has to be typed correctly.

You can find an introduction to error reporting here: https://gildor478.github.io/ounit/ounit2/index.html#error-reporting

You can find some options in assert_equal

assert_equal : [...] ?⁠printer:('a -> string) -> ?⁠pp_diff:[....]-> ?⁠msg:string -> [...]

• '~msg:"..."' will allow you to print a meaningful message
• '~printer' will allow you to simply print the value, for example "~printer:string_of_int"
• '~pp_diff:...' is a more advanced way to do comparison (have a look at https://gildor478.github.io/ounit/ounit2/OUnitDiff/index.html which provides an helper for that)

Here is an example of the output with ~msg and ~printer:

==============================================================================
Error: OUnit:6:OtherTests:0:TestFakeHTML.

File "/home/gildor/programmation/ounit/_build/default/test/oUnit-OUnit-pico#01.log", line 69, characters 1-1:
Error: OUnit:6:OtherTests:0:TestFakeHTML (in the log).

File "test/testOtherTests.ml", line 90, characters 1-1:
Error: OUnit:6:OtherTests:0:TestFakeHTML (in the code).

Number of test case in junit.xml.
expected: 7 but got: 6
------------------------------------------------------------------------------

[robmoore-i]

Ok, cheers. I'll close this since it seems to be a constraint from the language, so something more complex (I've tried to be useful and make a suggestion anyway) would be more difficult.

Given that constraint from ocaml, I wonder if assertions would be easier to do with a hamcrest-style (http://hamcrest.org/JavaHamcrest/tutorial), since it is further towards the composable end of the test-framework design spectrum. It'd use the type signature: assertThat : T -> T Matcher.t -> unit and you'd call it with something like assertThat "foo" (equalTo (myFunction "bar")).

Anyway, thanks for clarifying and giving additional info on how to improve the failure messages. Best wishes,

Rob