Test names as variables

sayanee commented 6 years ago

Issue

I'm trying to set test names as a variable so that it can be dynamic depending on the environment variable and test helper. But the test names are appearing to be all the same.

Steps to reproduce the error

Create a file test

#!/usr/bin/env bats

LS="expects to list files"
LSAL="expects to list files with details"

@test "$LS"  {
  COMMAND=$(ls)
  run $(COMMAND)
  [ "$status" -eq 0 ]
}

@test "$LSAL"  {
  COMMAND=$(ls -al)
  run $(COMMAND)
  [ "$status" -eq 0 ]
}

Execute this file test

$ ./test
 ✓ expects to list files with details
 ✓ expects to list files with details

2 tests, 0 failures

Actual result

Test names are all the same - specific they tend to take the name of the last variable

Expected result

Test names are all different.

Other options

I'm open to other options as long as I can set the test name dynamically. This is to cater for tests that can be different depending on the environment.

dimo414 commented 6 years ago

Note that the issue isn't just that the names are all the same - the last test is being run twice, and the first test isn't run at all.

This happens because the variables you've defined aren't in scope when BATS is preprocessing the test file. When the name is eval-ed LS and LSAL don't exist, so the resulting name for both tests is just test_, instead of test_expects_to_list_files and test_expects_to_list_files_with_details.

Workarounds

Just put a unique hard-coded string in each test name alongside the variables. Even just numbering them (@test "1: $LS", @test "2: $LSAL") would be sufficient.
You say the variables are dependent on the environment - if that's the case you could make them environment variables in the shell, rather than define them inside the .bats file. Environment variables (e.g. PWD, USER, RANDOM, etc.) will be properly eval-ed during preprocessing.

Fixes

The underlying issue is that BATS silently conflates tests with the same preprocessed name. It should probably fail if the encoded_name already exists in the tests array, rather than unilaterally adding it.

It might be sufficient to just include the index in the encoded_name - that would ensure every @test has a unique name.

- encoded_name="$(encode_name "$name")"
+ encoded_name="$index::$(encode_name "$name")"

@sstephenson - would you accept that patch?

sayanee commented 6 years ago

Just put a unique hard-coded string in each test name alongside the variables

This is exactly what I ended up doing.

Example of the tweaks based on the above code:

#!/usr/bin/env bats

LS="list files"
LSAL="with details"

@test "expects to $LS"  {
  COMMAND=$(ls)
  run $(COMMAND)
  [ "$status" -eq 0 ]
}

@test "expects to list $LSAL"  {
  COMMAND=$(ls -al)
  run $(COMMAND)
  [ "$status" -eq 0 ]
}

Output

$ ./test
 ✓ expects to list files
 ✓ expects to list files with details

2 tests, 0 failures

sayanee commented 6 years ago

closing this as the test name with the above tweaks are workable

sstephenson / bats