running a sub test returns 0 when the file does not exist

[cinterloper]

when running an internal test like @test "gen a client cert" { run bats test/pki/geninstallercrt.bats [ "$?" -eq 0 ] }

if test/pki/geninstallercrt.bats does not exist, the line still returns true, and appears to pass

[ztombol]

That's how it's supposed to work. From the Readme (emphasis is mine).

"Bats includes a run helper that invokes its arguments as a command, saves the exit status and output into special global variables, and then returns with a 0 status code so you can continue to make assertions in your test case."

You need to test $status instead of $?.

@test "gen a client cert" {
  run bats test/pki/geninstallercrt.bats
  [ "$status" -eq 0 ]
}

See the section on run for details and examples.

By the way, why do you want to run a test from a test?

[ztombol]

I'm assuming you posted a minimal example of the issue, and you do other things as well in that test function. Otherwise it wouldn't be a very useful test (and wouldn't actually need run).

[cinterloper]

Ok, yes i do things in the sub test, it creates an artifact that gets used in a later test

i was confused because , when executing a sub test that did not exist (the bats file is not there), i would expect an error to be thrown, not for it to pass

i ran into this issue because the artifact did not exist, but this block appeared to pass with a check mark

when looking closer, the bats file it self did not exist, yet this line appeared to pass with no error

[cinterloper]

this is the most minimal example:

% cat testunpresent.bats 
#!/usr/bin/env bats

@test "run a test that does not exist" {
  run bats notarealfile.bats
  [ "$?" -eq 0 ]
}
% bats/bin/bats testunpresent.bats
 ✓ run a test that does not exist

1 test, 0 failures

[ztombol]

Like I've said, this is the expected behaviour. run always returns 0. You should be testing $status.

In case you just want to test whether the sub-test returns success and nothing else, you don't even need run or the $status check. Even though a test like that would only make sense if you were testing Bats itself.

[cinterloper]

i dont think the behaviour is consistant with bats itself

% bats/bin/bats notafile.bats              
bats: /tmp/nothing/notafile.bats does not exist
% echo $?                                  
1

[ztombol]

I don't think you understand how Bats works. You should read the Readme again. Right at the beginning (emphasis is mine):

Test cases consist of standard shell commands. Bats makes use of Bash's errexit (set -e) option when running test cases. If every command in the test case exits with a 0 status code (success), the test passes. In this way, each line is an assertion of truth.

Bats' behaviour is well documented. Your test is wrong. I think, this is what you are trying to do:

$ cat test.bats 
@test "run a test that does not exist" {
  bats notarealfile.bats
}

$ bats test.bats 
 ✗ run a test that does not exist
   (in test file test.bats, line 2)
     `bats notarealfile.bats' failed
   bats: /path/to/notarealfile.bats does not exist

1 test, 1 failure

But I can't stress it enough that the test above makes no sense unless you are testing Bats itself or you add other commands too to the test.

[ztombol]

Additionally, if you just need the side effect from the sub-test (e.g. creating a file) then you should probably look at moving the side effect into a helper function and calling it from the test directly. I understand this isn't always the best solution (testing the helper may become more difficult), but it could simplify your test suite and even improve its performance (avoids launching a new Bats process).

It could look something like this:

#!/usr/bin/env bats

my_helper {
  touch '/path/to/file' && echo 'Success!'
}

# IMPORTANT: Test the helper before using it in other tests to make debugging them easier!
@test 'testing the helper' {
  run my_helper
  [ "$status" -eq 0 ]
  [ "$output" == 'Success!' ]
}

@test 'using the helper' {
  # Side-effect
  my_helper

  # Tests
  [ -f '/path/to/file' ]
  # more tests...
}