Unexpected behaviour of Scenario Outline

[leventyalcin]

Description : I'm trying to use the Tags example from the documentation. However, I get either wrong results or the tests results are incosistent between sequiential runs.

To Reproduce

1. It's a long one and I'll add required bits and pieces from the output of terraform show

2. terraform-compliance --features ./features/ --planfile ci.tfplan.json

3. Tried both python and docker executeable

4. Examples:
       | tags    | value        |
   ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
       Failure: Name property in module.vpc.aws_route53_zone.private resource does not match with ^[a-z0-9-]*$ case insensitive regex. It is set to vpc-devops.domain.internal.
       | Name    | ^[a-z0-9-]*$ |
         Failure:
   ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
       Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have service property.
       Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have service property.
       Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have service property.
       | service | ^[a-z0-9]*$  |
         Failure:
   ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
       Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have env property.
       Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have env property.
       Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have env property.
       | env     | ^(prod|uat)$ |
         Failure:
   ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
       Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have desc property.
       Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have desc property.
       Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have desc property.
       | desc    | .+           |
         Failure:

5. <Your feature/scenario/steps>

   Scenario Outline: Ensure that specific tags are defined
       Given I have resource that supports tags defined
       When it contains tags
       Then it must contain <tags>
       And its value must match the "<value>" regex
   
       Examples:
           | tags    | value         |
           | Name    | ^[a-z0-9-]*$  |
           | service | ^[a-z0-9]*$   |
           | env     | ^(prod\|uat)$ |
           | desc    | .+            |

Expected behavior : I've tried to few things and I'll try my best to explain.

---

First try This is my first attempt to use terraform-compliance and wanted to see how and if it really fails. Initially I was using the regex patters as

            | tags    | value         |
            | Name    | ^[a-z0-9]$  |
            | service | ^[a-z0-9]$   |
            | env     | ^(prod\|uat)$ |
            | desc    | .+            |

which means the value of Name and service tags only can be one alphanumeric character. However, I was getting the output before except the message was excatly the same with the other tags.

---

Second try

Then started the use the scenario

            | tags    | value         |
            | Name    | ^[a-z0-9-]*$  |
            | service | ^[a-z0-9]$   |
            | env     | ^(prod\|uat)$ |
            | desc    | .+            |

However, the message I got is this

    Examples:
        | tags    | value        |
    ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
        Failure: Name property in module.vpc.aws_route53_zone.private resource does not match with ^[a-z0-9-]*$ case insensitive regex. It is set to vpc-devops.domain.internal.
        | Name    | ^[a-z0-9-]*$ |
          Failure:
    ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
        Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have service property.
        Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have service property.
        Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have service property.
        | service | ^[a-z0-9]$   |
          Failure:
    ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
        Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have env property.
        Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have env property.
        Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have env property.
        | env     | ^(prod|uat)$ |
          Failure:
    ❗ WARNING: "When it contains tags" step functionality will be changed on future versions and the functionality will be same as "When it has tags" step. Please use the latter.
        Failure: module.vpc.aws_eip.ngw[0] (resource that supports tags) does not have desc property.
        Failure: module.vpc.aws_eip.ngw[1] (resource that supports tags) does not have desc property.
        Failure: module.vpc.aws_eip.ngw[2] (resource that supports tags) does not have desc property.
        | desc    | .+           |
          Failure:

while the value of

• expected: one alphanumeric character
• the real value: vpc

and there is 24 resources are having the service tag with the value vpc. I see no error for that apart from the aws_eip which has no tag at all.

---

On top of that; the regex for env is always ^(prod|uat)$ but never fails while the value in the state/plan is always devops

terraform show ci.tfplan | grep env
          + "env"            = "devops"
          + "env"            = "devops"
          + "env"            = "devops"
          + "env"            = "devops"
<multiple same lines>

Tested versions :

• terraform-compliance -v terraform-compliance v1.3.6 initiated
• terraform -v Terraform v0.13.5 + provider registry.terraform.io/hashicorp/aws v3.14.1
• python --version Python 3.8.5

[sleightsec]

@leventyalcin On the when clause, try using has instead of contains. The following works perfectly for me:

Scenario Outline: Required Tags are present on all resources which take tags
  Given I have resource that supports tags defined
  When it has tags
  Then it must contain <tags>
  And its value must match the "<value>" regex

  Examples:
    | tags        | value              |
    | name        | .+                 |
    | application | .+                 |
    | environment | ^(prod\|uat\|dev)$ |