assert_failure is not working well with piped commands

[alkhimey]

Due diligence

• [ ] I am familiar with the "Reporting issues" section of CONTRIBUTING.md
• [V] I have searched the existing issues and haven't found one that matches what I'm reporting or requesting now.
• [ ] If this is a bug report, I have searched the Bash changelog for information that may explain or provide clues to the behavior I'm observing and reference it in the body of the report.

Framework, Bash, and operating system version information

#!/usr/bin/env bats

load ../../testing/enviroment

@test "vanilla bats" {
   run $( ls | grep nonexist )
   [ "$status" -eq 0 ]
}

@test "using assert with file that foes not exists (should fail -> test should be successful" {
   run $( ls | grep nonexist )
   assert_failure
}

@test "using assert with file that foes exists (should not fail -> test fail" {
   run $( ls | grep t2.bats )
   assert_failure
}

Output:

 $ bats t2.bats
 ✓ vanilla bats
 ✗ using assert with file that foes not exists (should fail -> test should be successful
   (in test file t2.bats, line 14)
     `assert_failure' failed
   expected failure, but command succeeded
   STATUS: 0
   OUTPUT:

 ✓ using assert with file that foes exists (should not fail -> test fail

3 tests, 1 failure

Description

I copied the assertion scripts for Bats into my own project to use with Bats for testing command line. It seems that there is a problem with pipe commands. The result is opposite to what I expected (see code example).

[mbland]

Short answer/quick fix: Wrap your pipeline in a function or script, not in $().

This isn't a failure of the assertion; Bash evaluates the $() expression and passes the result to run, rather than passing the pipeline to run to execute.

In the $(ls | nonexistent) case, the expression evaluates to the empty string. run effectively has no arguments, resulting in a successful status. The equivalent effect can be observed on the command line:

$ $()
$ echo $?
0

In the $(ls | t2.bats) case, the expression evaluates to t2.bats, resulting in the command run t2.bats. Since t2.bats isn't executable, the run t2.bats command fails, and assert_failure passes.

[alkhimey]

Hi thank you for you reply. I had a feeling that it is something related to how bash works, rather with your function.

However, the suggestion of wrapping with a function does not work. Maybe I did it wrong? How does the function know when to return 0/1 status?

load ../../testing/enviroment

f() {
   ls | grep nonexist
}

@test "vanilla bats" {
   run $( ls | grep nonexist )
   [ "$status" -eq 0 ]
}

@test "vanilla bats should fail (file exists)" {
   run $( ls | grep t2.bats )
   [ "$status" -eq 0 ]
}

@test "using assert with file that does not exists (should fail -> test should be successful)" {
   run $( ls | grep nonexist )
   assert_failure
}

@test "using assert with file that does exists (should not fail -> test fail)" {
   run $( ls | grep t2.bats )
   assert_failure
}

@test "using assert on a function  with file that does not exists (should fail -> test should be successful)" {
   run f
   assert_failure
}

@test "using assert on a function with file that does exists (should not fail -> test fail)" {
   run f
   assert_failure
}

Ouput:

10:18 $ bats t2.bats
 ✓ vanilla bats
 ✗ vanilla bats should fail (file exists)
   (in test file t2.bats, line 18)
     `[ "$status" -eq 0 ]' failed
 ✗ using assert with file that does not exists (should fail -> test should be successful)
   (in test file t2.bats, line 24)
     `assert_failure' failed
   expected failure, but command succeeded
   STATUS: 0
   OUTPUT:

 ✓ using assert with file that does exists (should not fail -> test fail)
 ✓ using assert on a function  with file that does not exists (should fail -> test should be successful)
 ✓ using assert on a function with file that does exists (should not fail -> test fail)

6 tests, 2 failures

[mbland]

The last two test cases are calling the same function with the same arguments. Instead, try:

load ../../testing/enviroment

f() {
   ls | grep "$1"
}

@test "using assert on a function, nonexistent file (should fail -> test should be successful)" {
   run f 'nonexistent'
   assert_failure
}

@test "using assert on a function, existing file (should not fail -> test fail)" {
   run f 't2.bats'
   assert_failure
}

You should see:

$ bats t2.bats 
 ✓ using assert on a function, nonexistent file (should fail -> test should be successful)
 ✗ using assert on a function, existing file (should not fail -> test fail)
   (in test file t2.bats, line 14)
     `assert_failure' failed
   expected failure, but command succeeded
   STATUS: 0
   OUTPUT:
   t2.bats

2 tests, 1 failure

[alkhimey]

Oh, that was very careless of me. Thank you for the help :smile:

Is there a reason, you can think of, that this assert library can not be part of bats officially? I made a request for this in bats-core/bats-core#77

[mbland]

Just haven't gotten around to it yet is all. Even if it doesn't go into bats-core/bats-core, I'll probably post it as a separate repo somewhere. Nothing in that directory depends on anything in the rest of go-script-bash.