Test assertion and test output enhancements

Terraform Version

Terraform v1.9.5
on darwin_arm64

Use Cases

In the case of tests, having the error_message in the test assertions is not especially useful, and makes the tests slower to write. In a lot of cases, the assertion itself is pretty clear to understand. My proposal is to make error_message optional, and to add a description field to each test case, as well as to each assertion (with the understanding that tests may have many assertions).

Then, the terraform test output should show a dot for each assertion (similar to rspec's compact format), or the description of each assertion next to it in an expanded format (similar to mocha, pytest, rspec). If dynamic blocks or iteration are supported (as suggested here, there could maybe be a variable to provide the current value in the message as a formatting string.

If a description or error message isn't provided, the test could either provide just the output it provides on failure now, or it could provide a terser human readable summary of what the assertion is testing.

Having a configurable -output-format or -format or -output flag would also support things like generating xml reports or allowing GitHub style output for automagic annotation.

Attempted Solutions

n/a

Proposal

provider "google" {
  project = "foo-testproject"
}
run "basic" {
  description "Here is a basic test of secret manager related resources."

  command = plan
  plan_options {
    refresh = false
  }

  # Tests for secret itself
  assert {
    description = "Secret has the expected secret_id."
    condition    = google_secret_manager_secret.this.secret_id == "testsecret"
  }

  # Test permissions
  assert {
    description = "The expected number of IAM accessors are found."
    condition    = length(google_secret_manager_secret_iam_member.accessor_membership) == 4
  }

  assert {
    description = "The expected member is found."
    condition     = contains(
      keys(google_secret_manager_secret_iam_member.accessor_membership),
      "serviceAccount:bar@foo-testproject.iam.gserviceaccount.com"
    )
  }

Then running terraform test:

$ tf test
tests/main.tftest.hcl: pass
  run "basic"... pass ## one dot per assertion vs. always 3

or (exact formatting, command line flag, etc. could be different, just kind of an example)

$ tf test -output verbose
tests/main.tftest.hcl
    basic (Here is a basic test of secret manager related resources):
        Secret has the expected secret_id. ✔️
        The expected number of IAM accessors are found. ✔️
        The expected number of IAM accessors are found. ✔️
    other_test (Description):
[...]
Summary: 3/3 tests passed

Errors could be indicated on a line by line basis, and then more details provided at the end.

References

No response

hashicorp / terraform