Show real test cases' description when they are called from a test suite

[skyscribe]

Implicit test suite is supported (as least I suppose so) by wrapping another bats script, and load it in a top level test. This is smart, however below drawbacks hit my head

• Only one tests will be shown as executed, actually quite many of them are executed
• Only the outer test's name are shown, wrapped tests are all hidden
• Whenever there're failures inside the inner bats tests, it's not straightforward to figure out which case is wrong, only option is to check line number?

Things would be great if bats can support showing the details of what are executed, with better names, like combining the case names

[ztombol]

@skyscribe Please provide a minimal example. By any chance, is #116 what you are trying to do?

[skyscribe]

@ztombol Here's one simple example.

Dump full case name

Suppose I have a dummy.bats that defines below case:

@test "A dummy bats to bootstrap travis-ci" {
    run echo "ok"
    [ $status -eq 0 ]
}

In the outer suites.bats file

@test "dummy bootstrap suites" {
    run bats "dummy.bats"
    [ $status -eq 0 ]
}

Then a simple command bats suites.bats is meant to run all cases, the output would only show the name in suites.bats but the internal wrapped name.

✓ dummy bootstrap suites

It would be better if it can show like

✓ dummy bootstrap suites[A dummy bats to bootstrap travis-ci]

Confusing error prints?

If I modify the dummy case to make it fail by changing last checking ([ $status -eq 1]), then the output is a bit confusing:

 ✗ dummy bootstrap suites
   (in test file suites.bats, line 5)
     `[ $status -eq 0 ]' failed
     1..1
 ✗ dummy bootstrap suites
   (in test file dummy.bats, line 5)
     `[ $status -eq 1 ]' failed
     ok

In my opinion, it would be better to just show one single error than two.

[ztombol]

An implicit test suite, as you call it, is not intentionally supported. It's just happens to work. A Bats test can contain any Bash command. Running bats inside a test is not a special case.

Regardless, your output is definitely wrong. First, you should only see one error that is reported by suites.bats. Second, the two errors definitely shouldn't have the same description. If the test in suits.bats would print $output only then the error in dummy.bats, with its entire tap formatted output, would show through and would be displayed as comments (lines prefixed with #). Third, the format of the second error message is weird. There is a counter from the tap formatted output (1..1), but the message itself is in pretty format.

Can you try running suites.bats with bats --tap? That may give us a clearer picture of where the output is coming from. What OS and bash version are you running?

Here's what I got when running your example with [ "$status" -eq 1 ] in dummy.bats.

suites.bats:

#!/usr/bin/env bats

@test "dummy bootstrap suites" {
  run bats "dummy.bats"
  [ $status -eq 0 ]
}

dummy.bats:

#!/usr/bin/env bats

@test "A dummy bats to bootstrap travis-ci" {
  run echo "ok"
  [ $status -eq 1 ]
}

Output:

$ bats ./suites.bats 
 ✗ dummy bootstrap suites
   (in test file suites.bats, line 5)
     `[ $status -eq 0 ]' failed

1 test, 1 failure

Something is definitely wrong and we should get to the bottom of this. But there may be other/better ways of achieving what you what to do. I still don't understand the merit of running Bats from within Bats to start a test suite. (That's not saying that it doesn't have any!)

• Why not just run Bats on a directory containing all the test files? This is what people usually do in my experience.
• Maybe you don't want to run all test files in that directory. Then, why not just add bats test.file statements to a Bash script?
• Are you trying to share tests between suites? In that case, I still think you want #116.

Could you please elaborate?