Loop around tests?

[Thubo]

I would like to create the following scenario: (Note the for loop in the following code)

#!/usr/bin/env bats                                                                                          

set -o pipefail                                                                                              

###############################################################################                              
# Functions                                                                                                  
###############################################################################                              
free_space_in_gb() {                                                                                               
  echo $(df -P $1 | tail -n 1 | awk '{print int($4/1024/1024)}')                                             
}                                                                                                            

check_partition() {                                                                                          
  mount | grep " on $1 "                                                                                     
}                                                                                                            
###############################################################################                              

for i in /tmp /var; do                                                                                   
  @test "$i: exists" {                                                                                       
    stat $i  2>/dev/null
  }                                                                                                          
  @test "$i: separate partition" {                                                                           
    mount | grep " on $i "                                                                                   
  }                                                                                                          
done                                                                                                         

@test "/var: free space" {                                                                                   
  [[ $(free_space_in_gb /var) -gt 100 ]]                                                                           
}

Is there a simple way to achieve this?

The above example runs the tests only for the last variable in the for loop (here /var).

Maybe I'm on the wrong track here, but I have may very similar tests, which force me to create copy&paste code - which is something I would like to avoid....

On a side note: Is there a way to pass a parameter to a @test? I'm thinking about something like:

@test "run stat on $1" /var {
  stat $1 # Check if  /var is there
}

Thanks in advance!

[miroswan]

I'm having this same problem.

[ztombol]

This, as lengthy as it is, is not a definitive guide or a best practice. This is just an in-depth bisection of the problem and my shot at tackling it.

---

Why doesn't it work?

To understand why this doesn't work you need to know how Bats executes tests.

As you know, a test file is a Bash script with special syntax for defining test cases. Bats is not an interpreter, it doesn't know how to execute a test file. Instead, what it knows is how to translate the special syntax into valid Bash code.

This involves replacing @test commands with function headers, and adding other housekeeping code. The resulting Bash script is what is actually executed.

Diving deep

Let's see an example.

#!/usr/bin/env bats

for var in a b c; do
  @test "sample test ${var}" {
    run echo "${var}"
    [ "$output" == "$var" ]
  }
done

Contrary to what you'd expect, this does not define three different test cases. It defines only one. This becomes apparent when you take a look at the translated, or preprocessed, test file. To copy it before Bats deletes it, add the following snippet to the test file.

teardown() {
  dbg_save_source './bats-test.src'
}

# Save a copy of the preprocessed test file.
#
# Globals:
#   BATS_TEST_SOURCE
# Arguments:
#   $1 - [=./bats.$$.src] destination file/directory
# Returns:
#   none
dbg_save_source() {
  local -r dest="${1:-.}"
  cp --reflink=auto "$BATS_TEST_SOURCE" "$dest"
}

Now, when you run Bats, the teardown() function will save a copy of the preprocessed test file in ./bats-test.src.

Here it is with the helper functions removed for clarity.

#!/usr/bin/env bats

for var in a b c; do
test_sample_test_() { bats_test_begin "sample test ${var}" 10; 
    run echo "${var}"
    [ "$output" == "$var" ]
  }
done

# `teardown()' and `dbg_save_source()' were here...

bats_test_function test_sample_test_

The @test block is replaced with a function and some housekeeping code. The name of the function is derived from the test description.

But, as you can see, the loop variable ${var} was expanded to the empty string. This is because preprocessing is just simple text manipulation. The test file is scanned line-by-line and rewritten where necessary, but not actually executed. Therefore ${var} at this stage is undefined and thus expanded to the empty string.

Note: Variables in test descriptsions are expanded because the preprocessor puts it through eval. This is necessary to evaluate any escape sequences in the string.

When the preprocessed script is run, the for loop just defines the same test case (and therefore function) three times, and finally calls it once at the end. The loop variable keeps its final value, here c, after the loop terminates, and the only invocation of the test function runs with this value.

My solution

To avoid duplicating code in test cases, extract the common bits into a parametrised helper function and invoke that from each test with the appropriate parameters.

#!/usr/bin/env bats

helper() {
  local -r var="$1"
  run echo "${var}"
  [ "$output" == "$var" ]
}

@test "test \`a'" {
  helper a
}

@test "test \`b'" {
  helper b
}

@test "test \`c'" {
  helper c
}

Yes, you still need multiple @test blocks if you want a separate test case for each value. But the interesting bits, which are likely the longer and more complicated part of the tests, are centralised.

This encourages sharing code and developing reusable helper libraries, that further reduces maintenance. In fact this is how the bats-* family of libraries began.

For a real world example see this test file in Varrick.

FAQ

Can I pass a parameter to @test commands?

No. This is not only not supported by the preprocessor, but also doesn't make sense since you can't call a test case. Even if you could, where would you call the test cases? You would just push the problem one level up. You are on the right track though. Factor out the logic into a test helper and call that with different parameters from the test cases. See the proposed solution above.

Can't I just escape the variable in the description?

Nope. The description is encoded to make it suitable for use as a function name, e.g. white-space and other special characters are replaced. The $ ends up being encoded as well.

Can I use eval to generate my test cases?

No. Preprocessing doesn't care about what the code does. It simply manipulates text. Execution comes after this, and then it's already too late. You actually have to have physically separate @test blocks.

Can I write a script that generates the test file?

Yes, you can. But should you? This adds yet another piece of code that needs maintenance and testing. Your goal is to make your tests easier to read and maintain, not to reduce character count.

[Thubo]

Hi @ztombol,

thank you very much for the explanations and also for the links to the other projects. For now I helped myself by writing meta-code which generates the tests, executes them and does some cleanup. Next step is to look into your bats-docs project. I'll close the issue for now, but I hope your explanations will help others as well.

[arvenil]

#!/usr/bin/env bats

helper() {
  local -r var="$1"
  run echo "${var}"
  [ "$output" == "$var" ]
}

@test "test \`a'" {
  helper a
}

@test "test \`b'" {
  helper b
}

@test "test \`c'" {
  helper c
}

This doesn't seem to work

ubuntu@ubuntu-xenial:/vagrant$ bats loop.bats
 ✓ Run 1
 ✗ Run 2
   (in test file loop.bats, line 14)
     `[ "$status" -eq 0 ]' failed

2 tests, 1 failure
ubuntu@ubuntu-xenial:/vagrant$ cat loop.bats
#!/usr/bin/env bats

helper() {
    run bash -c 'echo a | grep b'
    [ "$status" -eq 0 ]
}

@test "Run 1" {
    run helper
}

@test "Run 2" {
    run helper
    [ "$status" -eq 0 ]
}

[arvenil]

nvm, I see my mistake :)

ubuntu@ubuntu-xenial:/vagrant$ cat loop.bats
#!/usr/bin/env bats

helper() {
    run bash -c 'echo a | grep b'
    [ "$status" -eq 0 ]
}

@test "Run 1" {
    helper
}
ubuntu@ubuntu-xenial:/vagrant$ bats loop.bats
 ✗ Run 1
   (from function `helper' in file loop.bats, line 5,
    in test file loop.bats, line 9)
     `helper' failed

1 test, 1 failure