Need more information in pretty output

[jayhendren]

If a test fails, the output looks like this:

foo.bash:

foo () {
  echo baz
}

foo_spec.bash:

source ./foo.bash
@test "foo equals bar" {
  expected_result="bar"
  run foo
  [ "$status" -eq 0 ]
  [ "$output" = "$expected_output" ]
}

[user@host ~]% bats foo_spec.bats
 ✗ foo equals bar
   (in test file foo_spec.bats, line 6)
     `[ "$output" = "$expected_output" ]' failed

2 tests, 1 failure, 1 skipped

bats lets me know which particular assertion failed, but it gives me no useful information about what the expected result was or what the actual result was. This makes it challenging and time-consuming to debug the function or script under test. In order to figure out that bats expected foo to output bar but received baz instead, I have to add printf/echo-style debugging statements to my script and/or my bats spec.

Compare this to, for instance, rspec:

foo.rb:

def foo()
  "baz"
end 

foo_spec.rb:

require_relative 'foo'

describe 'foo' do
  it 'should return "bar"' do
    actual_result = foo
    expected_result = 'bar'
    expect(actual_result).to eq(expected_result)
  end
end

[user@host ~]% rspec foo_spec.rb
F

Failures:

  1) foo should return "bar"
     Failure/Error: expect(actual_result).to eq(expected_result)

       expected: "bar"
            got: "baz"

       (compared using ==)
     # ./foo_spec.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.01638 seconds (files took 0.12804 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./foo_spec.rb:4 # foo should return "bar"

The rspec output clearly prints the expected value and the received value, making it easy to debug the failing function being tested. I can quickly and easily see that there is a typo in the return statement of the foo method.

[ztombol]

This is a duplicate of #190 and #191 (and many others).

[jayhendren]

Those were both closed by the user without a response from the developer. I'll keep this open pending a response from the developer.

[ztombol]

While I'm not familiar with RSpec, a quick look suggests that the bats-* libraries correspond well to RSpec gems. In fact, we named them similarly on purpose.

Your example with bats-assert would be:

@test "foo equals bar" {
  expected_result="bar"
  run foo
  assert_success
  assert_output "$expected_result"
}

Which would show you this:

$ bats foo_spec.bash 
 ✗ foo equals bar
   (from function `assert_output' in file test_helper/bats-assert/src/assert.bash, line 243,
    in test file foo_spec.bash, line 10)
     `assert_output "$expected_result"' failed

   -- output differs --
   expected : bar
   actual   : baz
   --

1 test, 1 failure

I'm assuming rspec-core alone can't give you useful information about failing assertions. It needs rspec-expectations to achieve that. Same goes for Bats. bats needs bats-assert.

You may need to wait quite a while for developer feedback as Bats is severly understaffed (#150). I got positive feedback from the developer during the early development of the bats-* libraries (#110). Thanks to the community, they just got better since then.

I'm not trying to sell anything. Just wanted to give a bit more insight on how this is done in Bats. Use whatever feels right for your task.

[jayhendren]

Ah, ok, thanks @ztombol. That really helps to clarify. You may be right about rspec-core and its relationship with the other gems; I've only ever used rspec as one bundle so I'm not sure how the individual gems are divided up. There doesn't seem to be any mention of the other libraries in the bats ecosystem in the README, wiki, or any other documentation I can find, but it sounds like the ones you've developed are what I'm looking for.

I'll keep this issue open in case the developer (or, future developer, if the project is transferred to a new dev) wishes to comment. However, it seems likely that the response will be to use bats + bats-assert.