setup() and teardown() are run on skipped tests

[michaelshobbs]

It would seem that tests that are set to skip, still get setup() and teardown() executed. Take this for example.

#!/usr/bin/env bats

setup() {
  sleep 30
}

teardown() {
  sleep 30
}

@test "ps" {
  skip "test setup/teardown on skip"
  run bash -c "echo hello"
  echo "output: "$output
  echo "status: "$status
  assert_success
}

# time bats test.bats
 - ps (skipped: test setup/teardown on skip)

1 test, 0 failures, 1 skipped

real    1m0.061s
user    0m0.013s
sys 0m0.038s

Is this expected behavior? Is it possible to make this not the case? Thanks for putting this project together. It's been great to work with!

[michaelshobbs]

bump!

[jake-low]

skip happens programmatically; it's a function that runs in the test, not an annotation. Consider this:

setup() {
  sleep 30
}

teardown() {
  sleep 30
}

@test "maybe skip" {
  if (( RANDOM % 2 )); then
    skip "flipped a coin, came up heads, so we're skipping"
  fi

  run echo "foo"
  [ $output = "foo" ]
}

setup has to run, because at that time we don't know whether the test will skip or not. teardown must therefore also run (to clean up whatever mess might've been made in setup).

[michaelshobbs]

Oh I totally get why it's not technically feasible in the current codebase. Are you saying that my proposed use case is invalid?

[jake-low]

Ah, I misunderstood.

I definitely think your use case is valid; other test frameworks that I've worked with implement skip as a test annotation. Pytest is a good example of this -- there's a decorator that you can apply to test functions which skips them. This means the fixtures they request (pytest tends to use fixtures rather than setup/teardown methods) are never created. In my experience, this is useful. For example, you can conditionally skip tests depending on, say, whether certain features of your program are configured. It'd be painful to set up fixtures for hundreds of tests that apply to a feature that's disabled.

I'll leave it to @sstephenson whether the use case is valid within the context of bats, or whether it could easily be implemented. I'm neither a bash expert nor a bats expert, just a random passer-by.

[max-k]

Hi, To skip you teardown, you can use something like:

if [[ "$BATS_TEST_SKIPPED" - eq 1 ]]; then
    return 0
fi

or

if [[ ! "$BATS_TEST_SKIPPED" ]]; then
    # do teardown here
fi

[ajmera25]

I am also facing above issues. I am making my test skipped in BeforeInvocation but in report I can see setUp and TearDown is marked as skipped but Test is marked as Failure?. Could you I know Why?.

Due to Which Final Status I can see like this.

MixedTests Total tests run: 1, Failures: 1, Skips: 0 Configuration Failures: 0, Skips: 2.

But I want test should show as Skip instead of Failure.