Data source checks are failing when being included in larger test suite

[japitts0889]

Description

When you have multiple features and scenarios defined that terraform-compliance is running through, it doesn't seem to be able to always find data types within the plans, but when running the feature/scenario on it's own it passes every time.

To Reproduce

1. Create a large number of features and scenarios and have terraform-compliance iterate through all of them.
2. Have a scenario that looks for a data source type
3. My particular one is checking for the aws_ami data source
4. It fails with the following error

Feature: 3-tier-web-benchmark-3_3  # /target/terraform-compliance/features/business_continuity/3-tier-web-benchmark-3_3.feature

    @noskip_at_line_5
    Scenario: Ensure Auto-Scaling Launch Configuration for Web-Tier is configured to use an approved Amazon Machine Image
        Given I have aws_launch_configuration defined
        Failure: Can not find aws_ami defined in target terraform plan.
        Given I have aws_ami data configured
          Failure: 
        Then it must have filter
        And its value must contain ami-name-*

5. But when I pull it out of the larger group and run the feature on its own against the same plan file it passes every time

Feature: 3-tier-web-benchmark-3_3  # /target/terraform-compliance/testing/3-tier-web-benchmark-3_3.feature

    @noskip_at_line_5
    Scenario: Ensure Auto-Scaling Launch Configuration for Web-Tier is configured to use an approved Amazon Machine Image
        Given I have aws_launch_configuration defined
        Given I have aws_ami data configured
        Then it must have filter
        And its value must contain ami-name-*

6. The only parameters i'm running with are the -f for the feature directory and -p. Also using -d for debugging trying to figure out why it's not finding it.

7. My full test suite has the following amount of features, scenarios, and steps in it. The only 2 that are failing are from these data source checks

   51 features (28 passed, 2 failed, 21 skipped)
   82 scenarios (46 passed, 2 failed, 34 skipped)
   331 steps (212 passed, 2 failed, 38 skipped)

8. The file structure of the terraform is fairly simple. The data calls are on the root of the terraform config and not within a sub module or anything \

9. The data call is simple as well. Just looking for an ami with a filter against the name and account id as shown below. I unfortunately can't provide the full terraform config or the plan file due to restriction.

data "aws_ami" "base" {
  owners      = [var.aws_account_id]
  most_recent = true

  filter {
    name   = "owner-id"
    values = [var.aws_account_id]
  }

  filter {
    name   = "name"
    values = ["ami-name-*"]
  }
}

Feature File:

Feature: 3-tier-web-benchmark-3_3
    @noskip_at_line_5
    Scenario: Ensure Auto-Scaling Launch Configuration for Web-Tier is configured to use an approved Amazon Machine Image
        Given I have aws_launch_configuration defined
        Given I have aws_ami data configured
        Then it must have filter
        And its value must contain ami-name-*

Sample Terraform Code:

data "aws_ami" "base" {
  owners      = [var.aws_account_id]
  most_recent = true

  filter {
    name   = "owner-id"
    values = [var.aws_account_id]
  }

  filter {
    name   = "name"
    values = ["ami-name-*"]
  }
}

Used terraform-compliance Parameters:

Just using -d for debugging

Running via Docker:

Yes

Error Output:

`Feature: 3-tier-web-benchmark-3_3  # /target/terraform-compliance/features/business_continuity/3-tier-web-benchmark-3_3.feature

    @noskip_at_line_5
    Scenario: Ensure Auto-Scaling Launch Configuration for Web-Tier is configured to use an approved Amazon Machine Image
        Given I have aws_launch_configuration defined
        Failure: Can not find aws_ami defined in target terraform plan.
        Given I have aws_ami data configured
          Failure: 
        Then it must have filter
        And its value must contain ami-name-*

Expected Behavior: The feature to pass or act the same as it does when running it individually

Tested Versions:

• terraform-compliance version: 1.3.24 & 1.3.22
• terraform version: 0.12.31
• python version: running in docker

[japitts0889]

Doing some more testing it does seem that it even happens when only using 2 feature files. I can provide what i'm doing below.

Feature # 1

@3_tier_web_3
Feature: 3-tier-web-benchmark-3_1
    @3_tier_web_3_1
    @noskip_at_lines_8_9
    @exclude_module.gitaly.aws_autoscaling_group.gitaly
    Scenario: Ensure each Auto-Scaling Group has an associated Elastic Load Balancer
        Given I have aws_autoscaling_group resources configured
        Then it must contain load_balancers
        And its value must not be null

Feature # 2

@3_tier_web_3
Feature: 3-tier-web-benchmark-3
    @3_tier_web_3_3
    @noskip_at_line_7
    Scenario: Ensure Auto-Scaling Launch Configuration for Web-Tier is configured to use an approved Amazon Machine Image
        Given I have aws_launch_configuration defined
        Given I have aws_ami data configured
        Then it must have filter
        And its value must contain ami-name-*

When running Feater # 2 on it's own it runs perfectly fine and passes every time. When running it together with feature #1 it will fail saying it can't find aws_ami

I've tested this by both having these 2 feature files in a directory with nothing else, as well as running them by using the tags associated with both features @3_tier_web_3

Command used to run:

terraform-compliance -f terraform-compliance/features/ -p tfplans/my_plan_name.json --tags 3_tier_web_3

[eerkunt]

Thanks for the bug report 🎉 Is it possible to post a plan.output ? I guess the terraform code snippet you wrote won't be enough to reproduce this scenarios. We can also write it by ourselves, but want to be sure if there is an edge case within your problem.

[japitts0889]

Unfortunately I can't. The plan is quite large at 604705 lines and there's a number of things that would have to be redacted from it to be able to post anything.

I can say though that the same issue does happen for any plan file I've been able to run it against so far.

[japitts0889]

So I was able to create a very simple test to be able to recreate this for you.

Terraform:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "example" {
  ami           = data.aws_ami.example.id
  instance_type = "t3.micro"
  tags = {
    Name = "HelloWorld"
  }
}

data "aws_ami" "example" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn-ami-hvm-*"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

Feature1:

@test1
Feature: Test 1 to include a data source check
    Scenario: This test will include a check for the data source aws_ami
        Given I have aws_ami data configured
        Then it must have filter

Feature2:

@test2
Feature: Test 2 just to have multiple feature files defined
    Scenario: Test 2 to check for taggingn
        Given I have resource that supports tags defined
        Then it must have tags

With this configuration when running both features it will never find the aws_ami data source and will always show the following output:

@test2
Feature: Test 2 just to have multiple feature files defined  # /Users/jessepitts/projects/terraform-compliance/cli/tests/functional/test_issue_534/test2.feature

    Scenario: Test 2 to check for taggingn
        Given I have resource that supports tags defined
        Then it must have tags

@test1
Feature: Test 1 to include a data source check  # /Users/jessepitts/projects/terraform-compliance/cli/tests/functional/test_issue_534/test1.feature

    Scenario: This test will include a check for the data source aws_ami
    💡 SKIPPING: Can not find aws_ami defined in target terraform plan.
        Given I have aws_ami data configured
        Then it must have filter

When I test just against the feature that is looking for the aws_ami with this command terraform-compliance -f ./ -p tfplan.json --tags test1 it works just as it should and passes the feature

@test1
Feature: Test 1 to include a data source check  # /Users/jessepitts/projects/terraform-compliance/cli/tests/functional/test_issue_534/test1.feature

    Scenario: This test will include a check for the data source aws_ami
        Given I have aws_ami data configured
        Then it must have filter

1 features (1 passed)
1 scenarios (1 passed)
2 steps (2 passed)

[japitts0889]

Here is the json output from the tfplan that i'm testing against.

{"format_version":"0.2","terraform_version":"1.0.5","planned_values":{"root_module":{"resources":[{"address":"aws_instance.example","mode":"managed","type":"aws_instance","name":"example","provider_name":"registry.terraform.io/hashicorp/aws","schema_version":1,"values":{"ami":"ami-03e8f862f1241c4ed","credit_specification":[],"get_password_data":false,"hibernation":null,"iam_instance_profile":null,"instance_type":"t3.micro","launch_template":[],"source_dest_check":true,"tags":{"Name":"HelloWorld"},"tags_all":{"Name":"HelloWorld"},"timeouts":null,"volume_tags":null},"sensitive_values":{"capacity_reservation_specification":[],"credit_specification":[],"ebs_block_device":[],"enclave_options":[],"ephemeral_block_device":[],"ipv6_addresses":[],"launch_template":[],"metadata_options":[],"network_interface":[],"root_block_device":[],"secondary_private_ips":[],"security_groups":[],"tags":{},"tags_all":{},"vpc_security_group_ids":[]}}]}},"resource_changes":[{"address":"aws_instance.example","mode":"managed","type":"aws_instance","name":"example","provider_name":"registry.terraform.io/hashicorp/aws","change":{"actions":["create"],"before":null,"after":{"ami":"ami-03e8f862f1241c4ed","credit_specification":[],"get_password_data":false,"hibernation":null,"iam_instance_profile":null,"instance_type":"t3.micro","launch_template":[],"source_dest_check":true,"tags":{"Name":"HelloWorld"},"tags_all":{"Name":"HelloWorld"},"timeouts":null,"volume_tags":null},"after_unknown":{"arn":true,"associate_public_ip_address":true,"availability_zone":true,"capacity_reservation_specification":true,"cpu_core_count":true,"cpu_threads_per_core":true,"credit_specification":[],"disable_api_termination":true,"ebs_block_device":true,"ebs_optimized":true,"enclave_options":true,"ephemeral_block_device":true,"host_id":true,"id":true,"instance_initiated_shutdown_behavior":true,"instance_state":true,"ipv6_address_count":true,"ipv6_addresses":true,"key_name":true,"launch_template":[],"metadata_options":true,"monitoring":true,"network_interface":true,"outpost_arn":true,"password_data":true,"placement_group":true,"primary_network_interface_id":true,"private_dns":true,"private_ip":true,"public_dns":true,"public_ip":true,"root_block_device":true,"secondary_private_ips":true,"security_groups":true,"subnet_id":true,"tags":{},"tags_all":{},"tenancy":true,"user_data":true,"user_data_base64":true,"vpc_security_group_ids":true},"before_sensitive":false,"after_sensitive":{"capacity_reservation_specification":[],"credit_specification":[],"ebs_block_device":[],"enclave_options":[],"ephemeral_block_device":[],"ipv6_addresses":[],"launch_template":[],"metadata_options":[],"network_interface":[],"root_block_device":[],"secondary_private_ips":[],"security_groups":[],"tags":{},"tags_all":{},"vpc_security_group_ids":[]}}}],"prior_state":{"format_version":"0.2","terraform_version":"1.0.5","values":{"root_module":{"resources":[{"address":"data.aws_ami.example","mode":"data","type":"aws_ami","name":"example","provider_name":"registry.terraform.io/hashicorp/aws","schema_version":0,"values":{"architecture":"x86_64","arn":"arn:aws:ec2:us-east-1::image/ami-03e8f862f1241c4ed","block_device_mappings":[{"device_name":"/dev/xvda","ebs":{"delete_on_termination":"true","encrypted":"false","iops":"0","snapshot_id":"snap-023c3e5a00f2024df","throughput":"0","volume_size":"8","volume_type":"gp2"},"no_device":"","virtual_name":""}],"creation_date":"2021-07-22T22:55:54.000Z","description":"Amazon Linux AMI 2018.03.0.20210721.0 x86_64 HVM gp2","ena_support":true,"executable_users":null,"filter":[{"name":"name","values":["amzn-ami-hvm-*"]},{"name":"root-device-type","values":["ebs"]},{"name":"virtualization-type","values":["hvm"]}],"hypervisor":"xen","id":"ami-03e8f862f1241c4ed","image_id":"ami-03e8f862f1241c4ed","image_location":"amazon/amzn-ami-hvm-2018.03.0.20210721.0-x86_64-gp2","image_owner_alias":"amazon","image_type":"machine","kernel_id":null,"most_recent":true,"name":"amzn-ami-hvm-2018.03.0.20210721.0-x86_64-gp2","name_regex":null,"owner_id":"137112412989","owners":["amazon"],"platform":null,"platform_details":"Linux/UNIX","product_codes":[],"public":true,"ramdisk_id":null,"root_device_name":"/dev/xvda","root_device_type":"ebs","root_snapshot_id":"snap-023c3e5a00f2024df","sriov_net_support":"simple","state":"available","state_reason":{"code":"UNSET","message":"UNSET"},"tags":{},"usage_operation":"RunInstances","virtualization_type":"hvm"},"sensitive_values":{"block_device_mappings":[{"ebs":{}}],"filter":[{"values":[false]},{"values":[false]},{"values":[false]}],"owners":[false],"product_codes":[],"state_reason":{},"tags":{}}}]}}},"configuration":{"provider_config":{"aws":{"name":"aws","expressions":{"region":{"constant_value":"us-east-1"}}}},"root_module":{"resources":[{"address":"aws_instance.example","mode":"managed","type":"aws_instance","name":"example","provider_config_key":"aws","expressions":{"ami":{"references":["data.aws_ami.example.id","data.aws_ami.example"]},"instance_type":{"constant_value":"t3.micro"},"tags":{"constant_value":{"Name":"HelloWorld"}}},"schema_version":1},{"address":"data.aws_ami.example","mode":"data","type":"aws_ami","name":"example","provider_config_key":"aws","expressions":{"filter":[{"name":{"constant_value":"name"},"values":{"constant_value":["amzn-ami-hvm-*"]}},{"name":{"constant_value":"root-device-type"},"values":{"constant_value":["ebs"]}},{"name":{"constant_value":"virtualization-type"},"values":{"constant_value":["hvm"]}}],"most_recent":{"constant_value":true},"owners":{"constant_value":["amazon"]}},"schema_version":0}]}}}