Allow custom failure messages

[ajacksified]

Via email:

In the unit testing frameworks I am familiar with, there is usually functionality for custom output upon failure of an assertion. I am finding that with my busted assertions, often I am unsure of what input caused the failure, as I perform a lot of operations like this:

for i = 1, 19 do
  local truncated = thingy:pulseTruncate( i )
  assert.are.equal( 0, truncated )
end

With the default busted output, I am able to see the invalid return value that causes the problem, but not which input value caused the problem. Normally I'd do something like this (pseudocode)

for i = 1, 19 do
  local truncated = thingy:pulseTruncate( i )
  assert.are.equal({
    expect = 0,
    actual = truncated,
    fail = function()
      print("Failed on $i, value of $truncated")
    end
  })
end

Is this possible with busted? I was unable to find functionality like this in the documentation or by reviewing the code.

[ajacksified]

A few options I've thought of:

assert.message("Failed on $i, value of $truncated").are.equal(0, i)

assert.onFailure(someFunction).are.equal(0, i)

[ajacksified]

https://github.com/Olivine-Labs/luassert#customized-formatters

[o-lim]

@ajacksified how would you feel about changing the assert functions to take an optional failure message argument? This would mean that assert.are.same and assert.are.equal would have to take 2 arguments plus an optional failure message, instead of comparing an arbitrary number of arguments.

This would be an API change, but it would also conform more to the standard Lua assert interface of

assert(boolean, "error message")

and you could do stuff like

assert.are.equal(0, values[i], "Failed at index " .. i)
assert.is_table(test[i], "Element at index  " .. i .. " is not a table")
assert.are.same(t0, test[i], "Failed at index " .. i)
assert.is_string(msg[i], "Failed at index " .. i)

The final error message when a custom failure message is specified could be of the format:

test_spec.lua:5: Custom Failure Message Here
Expected objects to be the same.
Passed in:
(table): { }
Expected:
(table): { ... }

otherwise, you would just get the standard error message:

test_spec.lua:5: Expected objects to be the same.
Passed in:
(table): { }
Expected:
(table): { ... }

Most people may not be calling assert.are.equal and assert.are.same with more than 2 arguments anyways, so this may not be as significant a change as one might think.

To me this seems like the cleanest interface to customize failure messages. Using customized formatters doesn't really do the trick because they only work on transforming the output format of an argument to the assert functions and not the failure message itself.

What are your thoughts? I'm thinking about submitting a PR that would do this, but I don't want to do it if you think it is a bad idea.