Terraform test - expect_failures does not display the validation rule error message in verbose mode

Terraform Version

1.7.4

Use Cases

When I run a test and want to test the validation rules are working as expected I can set some fake attribute values to force the validation rule to fail. This is expected but when I add the expect_failures = [xxx] the terraform logs don't display the validation rules that has failed. keeping those validation rules with the failure message will put an evidence the rule has failed as expected.

Attempted Solutions

Here a test run [expect_failures commented]:

run "validation_rules_fails" {
  command = plan

  variables {
    name                = "demo"
    resource_group_name = "demo"
    location            = "southeastasia"
    sku = {
      name     = "Dev(No SLA)_Standard_D11_v2"
      capacity = 1
    }

    kusto_cluster_principal_assignments = {
      rbac1 = {
        name                = "KustoPrincipalAssignment"
        resource_group_name = run.setup_dependencies.azurerm_resource_group.name
        principal_id        = run.setup_dependencies.client_config.client_id
        tenant_id           = run.setup_dependencies.client_config.tenant_id
        principal_type      = "Fake"
        role                = "Unknown"
      }
    }
  }

  # expect_failures = [
  #   var.kusto_cluster_principal_assignments
  # ]
}

result:

run "validation_rules_fails"... fail
╷
│ Error: Invalid value for variable
│ 
│   on variables.tf line 309:
│  309: variable "kusto_cluster_principal_assignments" {
│     ├────────────────
│     │ var.kusto_cluster_principal_assignments is map of object with 1 element
│ 
│ Only the following values are authorised: 'App', 'Group' or 'User'. Fix the value you have set to: [Fake]
│ 
│ This was checked by the validation rule at variables.tf:329,3-13.
╵
╷
│ Error: Invalid value for variable
│ 
│   on variables.tf line 309:
│  309: variable "kusto_cluster_principal_assignments" {
│     ├────────────────
│     │ var.kusto_cluster_principal_assignments is map of object with 1 element
│ 
│ Only the following values are authorised: 'AllDatabasesAdmin' or 'AllDatabasesViewer'. Fix the value you have set to: [Unknown]
│ 
│ This was checked by the validation rule at variables.tf:345,3-13.
╵
tests/unit_tests_cluster_principal_assignment.tftest.hcl... tearing down
tests/unit_tests_cluster_principal_assignment.tftest.hcl... fail

Failure! 2 passed, 1 failed.

Now if I set the expect_failures and re-run:

run "validation_rules_fails" {
  command = plan

  variables {
    name                = "demo"
    resource_group_name = "demo"
    location            = "southeastasia"
    sku = {
      name     = "Dev(No SLA)_Standard_D11_v2"
      capacity = 1
    }

    kusto_cluster_principal_assignments = {
      rbac1 = {
        name                = "KustoPrincipalAssignment"
        resource_group_name = run.setup_dependencies.azurerm_resource_group.name
        principal_id        = run.setup_dependencies.client_config.client_id
        tenant_id           = run.setup_dependencies.client_config.tenant_id
        principal_type      = "Fake"
        role                = "Unknown"
      }
    }
  }

  expect_failures = [
    var.kusto_cluster_principal_assignments
  ]
}

I get the following result:

 run "validation_rules_fails"... pass

Planning failed. Terraform encountered an error while generating this plan.

tests/unit_tests_cluster_principal_assignment.tftest.hcl... tearing down
tests/unit_tests_cluster_principal_assignment.tftest.hcl... pass

Success! 3 passed, 0 failed.

Proposal

My proposition is to get a combined view of the validation rule errors:

Expected:

run "validation_rules_fails" {
  command = plan

  variables {
    name                = "demo"
    resource_group_name = "demo"
    location            = "southeastasia"
    sku = {
      name     = "Dev(No SLA)_Standard_D11_v2"
      capacity = 1
    }

    kusto_cluster_principal_assignments = {
      rbac1 = {
        name                = "KustoPrincipalAssignment"
        resource_group_name = run.setup_dependencies.azurerm_resource_group.name
        principal_id        = run.setup_dependencies.client_config.client_id
        tenant_id           = run.setup_dependencies.client_config.tenant_id
        principal_type      = "Fake"
        role                = "Unknown"
      }
    }
  }

  expect_failures = [
    var.kusto_cluster_principal_assignments
  ]
}

terraform test --verbose

run "validation_rules_fails"... pass.  <<<---- The status is set to pass
╷
│ Error: Invalid value for variable.
│ 
│   on variables.tf line 309:
│  309: variable "kusto_cluster_principal_assignments" {
│     ├────────────────
│     │ var.kusto_cluster_principal_assignments is map of object with 1 element
│ 
│ Only the following values are authorised: 'App', 'Group' or 'User'. Fix the value you have set to: [Fake]
│ 
│ This was checked by the validation rule at variables.tf:329,3-13.
╵
╷
│ Error: Invalid value for variable
│ 
│   on variables.tf line 309:
│  309: variable "kusto_cluster_principal_assignments" {
│     ├────────────────
│     │ var.kusto_cluster_principal_assignments is map of object with 1 element
│ 
│ Only the following values are authorised: 'AllDatabasesAdmin' or 'AllDatabasesViewer'. Fix the value you have set to: [Unknown]
│ 
│ This was checked by the validation rule at variables.tf:345,3-13.
╵
Planning failed. Terraform encountered an error while generating this plan, however it was expected. <<<--- add an explanation.

tests/unit_tests_cluster_principal_assignment.tftest.hcl... tearing down
tests/unit_tests_cluster_principal_assignment.tftest.hcl... pass

Success! 3 passed, 0 failed.

References

No response

hashicorp / terraform