search

aws-ia / terraform-aws-control_tower_account_factory

AWS Control Tower Account Factory
Apache License 2.0
604 stars 386 forks source link

Upgrading AFT directly from 1.9.0 --> 1.11.1 results in import failures for `aft_account_provisioning_framework_account_metadata_ssm` #409

Closed matthewbarreiro closed 6 months ago

matthewbarreiro commented 6 months ago

Note: I deviated slightly from the issue template in an attempt to improve readability.

Terraform Version & Prov

AFT Version: 1.9.0 and 1.11.1 (with 1.10.4 used as an intermediate step for debugging)

Terraform Version & Provider Versions

AFT @ 1.9.0

Summary:

AFT Module    = 1.9.0
Terraform     = 1.3.6
AWS Provider  = 4.48.0
Full Details - Click to expand for full details and CLI output Note: To get this output, I downgraded my local env from AFT `1.11.1` to `1.9.0` `terraform version` ```shell Terraform v1.3.6 on darwin_arm64 + provider registry.terraform.io/hashicorp/archive v2.4.0 + provider registry.terraform.io/hashicorp/aws v4.48.0 + provider registry.terraform.io/hashicorp/local v2.4.0 + provider registry.terraform.io/hashicorp/random v3.6.0 + provider registry.terraform.io/hashicorp/time v0.10.0 ``` `terraform providers` ```shell Providers required by configuration: . ├── provider[registry.terraform.io/hashicorp/aws] 4.48.0 └── module.control_tower_account_factory ├── provider[registry.terraform.io/hashicorp/local] ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0, < 5.0.0 ├── module.aft_feature_options │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 ├── module.aft_iam_roles │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.aft_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.audit_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.audit_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.ct_management_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.ct_management_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.log_archive_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ ├── module.log_archive_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── module.aft_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 ├── module.aft_account_provisioning_framework │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 ├── module.aft_code_repositories │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── provider[registry.terraform.io/hashicorp/local] ├── module.aft_customizations │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── provider[registry.terraform.io/hashicorp/local] ├── module.aft_backend │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 ├── module.packaging │ └── provider[registry.terraform.io/hashicorp/archive] ├── module.aft_account_request_framework │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── provider[registry.terraform.io/hashicorp/time] ├── module.aft_lambda_layer │ ├── provider[registry.terraform.io/hashicorp/random] │ ├── provider[registry.terraform.io/hashicorp/local] │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 └── module.aft_ssm_parameters ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 └── provider[registry.terraform.io/hashicorp/random] Providers required by state: provider[registry.terraform.io/hashicorp/random] provider[registry.terraform.io/hashicorp/aws] provider[registry.terraform.io/hashicorp/local] provider[registry.terraform.io/hashicorp/time] provider[registry.terraform.io/hashicorp/archive] ```

AFT @ 1.10.4

Summary:

AFT Module    = 1.10.4
Terraform     = 1.3.6
AWS Provider  = 4.66.0
Full Details - Click to expand for full details and CLI output `terraform version` ```shell Terraform v1.3.6 on darwin_arm64 + provider registry.terraform.io/hashicorp/archive v2.4.0 + provider registry.terraform.io/hashicorp/aws v4.66.0 + provider registry.terraform.io/hashicorp/local v2.4.0 + provider registry.terraform.io/hashicorp/random v3.6.0 + provider registry.terraform.io/hashicorp/time v0.10.0 ``` `terraform providers` ```shell Providers required by configuration: . ├── provider[registry.terraform.io/hashicorp/aws] 4.66.0 └── module.control_tower_account_factory ├── provider[registry.terraform.io/hashicorp/local] ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 ├── module.aft_code_repositories │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ └── provider[registry.terraform.io/hashicorp/local] ├── module.aft_ssm_parameters │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ └── provider[registry.terraform.io/hashicorp/random] ├── module.aft_account_provisioning_framework │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 ├── module.aft_backend │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 ├── module.aft_customizations │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ └── provider[registry.terraform.io/hashicorp/local] ├── module.aft_feature_options │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 ├── module.aft_iam_roles │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 │ ├── module.log_archive_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.aft_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.aft_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.audit_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.audit_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.ct_management_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── module.ct_management_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ └── module.log_archive_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 ├── module.aft_account_request_framework │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── provider[registry.terraform.io/hashicorp/time] ├── module.aft_lambda_layer │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0 │ ├── provider[registry.terraform.io/hashicorp/random] │ └── provider[registry.terraform.io/hashicorp/local] └── module.packaging └── provider[registry.terraform.io/hashicorp/archive] Providers required by state: provider[registry.terraform.io/hashicorp/random] provider[registry.terraform.io/hashicorp/local] provider[registry.terraform.io/hashicorp/time] provider[registry.terraform.io/hashicorp/aws] provider[registry.terraform.io/hashicorp/archive] ```

AFT @ 1.11.1

Summary:

AFT Module    = 1.11.1
Terraform     = 1.3.6
AWS Provider  = 5.11.0
Full Details - Click to expand for full details and CLI output `terraform version` ```shell Terraform v1.3.6 on darwin_arm64 + provider registry.terraform.io/hashicorp/archive v2.4.0 + provider registry.terraform.io/hashicorp/aws v5.11.0 + provider registry.terraform.io/hashicorp/local v2.4.0 + provider registry.terraform.io/hashicorp/random v3.6.0 + provider registry.terraform.io/hashicorp/time v0.10.0 ``` `terraform providers` ```shell Providers required by configuration: . ├── provider[registry.terraform.io/hashicorp/aws] 5.11.0 └── module.control_tower_account_factory ├── provider[registry.terraform.io/hashicorp/local] ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 ├── module.aft_ssm_parameters │ ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ └── provider[registry.terraform.io/hashicorp/random] ├── module.aft_backend │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 ├── module.packaging │ └── provider[registry.terraform.io/hashicorp/archive] ├── module.aft_customizations │ ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ └── provider[registry.terraform.io/hashicorp/local] ├── module.aft_feature_options │ └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 ├── module.aft_account_provisioning_framework │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 ├── module.aft_account_request_framework │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0 │ └── provider[registry.terraform.io/hashicorp/time] ├── module.aft_iam_roles │ ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0 │ ├── module.aft_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.audit_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.audit_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.ct_management_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.ct_management_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.log_archive_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ ├── module.log_archive_service_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ └── module.aft_exec_role │ └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 ├── module.aft_code_repositories │ ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 │ └── provider[registry.terraform.io/hashicorp/local] └── module.aft_lambda_layer ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0 ├── provider[registry.terraform.io/hashicorp/random] └── provider[registry.terraform.io/hashicorp/local] Providers required by state: provider[registry.terraform.io/hashicorp/aws] provider[registry.terraform.io/hashicorp/archive] provider[registry.terraform.io/hashicorp/time] provider[registry.terraform.io/hashicorp/local] provider[registry.terraform.io/hashicorp/random] ```

Bug Description

After upgrading from AFT 1.9.0 directly to 1.11.1, aft_account_provisioning_framework_account_metadata_ssm fails with the following error: Unable to import module 'aft_account_provisioning_framework_account_metadata_ssm': No module named 'aft_common.constants

Downgrading back to 1.9.0, then upgrading from 1.9.0 --> 1.10.4 --> 1.11.1 seems to clear this problem.

To Reproduce

Steps to reproduce the behavior:

  1. Upgrade from 1.9.0 directly to 1.11.1
  2. Invoke aft-invoke-customizations step function on one or more accounts. Import error occurs.

Expected behavior

Upgrading from 1.9.0 --> >=1.11.1 should either work as expected, OR a note should be added indicating users must upgrade to 1.10.4 before moving to >=1.11.1.

Related Logs

Logs from aft_account_provisioning_framework_account_metadata_ssm as shown in the Step Functions console:

{
  "resourceType": "lambda",
  "resource": "invoke",
  "error": "Runtime.ImportModuleError",
  "cause": {
    "errorMessage": "Unable to import module 'aft_account_provisioning_framework_account_metadata_ssm': No module named 'aft_common.constants'",
    "errorType": "Runtime.ImportModuleError",
    "requestId": "REMOVED",
    "stackTrace": []
  }
}

Additional context

I was able to work around this issue by downgrading back to 1.9.0, then upgrading from 1.9.0 --> 1.10.4 --> 1.11.1. Full details, including config changes, follow in collapsed section.

Note: I'm assuming the value of terraform_version specified in the module configuration should in no way effect the behavior of the Step Functions / Lambdas functions (at least with regards to this specific issue). But for completeness, I am including my changes to that value in the steps below.

  1. Upgrade from 1.9.0 directly to 1.11.1

  2. Invoke aft-invoke-customizations step function, error occurs

  3. Downgrade back to 1.9.0, using the same terraform_version, AWS provider, and AFT versions used originally (all pinned to exact versions in my main.tf).

    Terraform HCL for AFT @ 1.9.0 ```hcl module "control_tower_account_factory" { source = "aws-ia/control_tower_account_factory/aws" version = "1.9.0" aft_feature_cloudtrail_data_events = false aft_feature_delete_default_vpcs_enabled = true aft_management_account_id = "REMOVED" aft_metrics_reporting = false aft_vpc_endpoints = false audit_account_id = "REMOVED" ct_home_region = "us-east-1" ct_management_account_id = "REMOVED" log_archive_account_id = "REMOVED" terraform_distribution = "oss" terraform_version = "1.3.6" tf_backend_secondary_region = "us-west-2" vcs_provider = "codecommit" } terraform { backend "s3" { bucket = "REMOVED" key = "aws-control-tower-aft/terraform.tfstate" region = "us-east-1" } required_providers { aws = { source = "hashicorp/aws" version = "4.48.0" } } } provider "aws" { region = "us-east-1" allowed_account_ids = ["REMOVED"] #management account default_tags { tags = { CostCenter = "AFT" } } } ```

  4. Invoked aft-invoke-customizations step function on a single account, completes successfully

  5. Upgrade AFT 1.9.0 --> 1.10.4, with provider version 4.48.0 --> 4.66.0 and no change to terraform_version in main.tf (1.3.6)

    Terraform HCL for AFT @ 1.10.4 ```hcl module "control_tower_account_factory" { source = "aws-ia/control_tower_account_factory/aws" version = "1.10.4" aft_feature_cloudtrail_data_events = false aft_feature_delete_default_vpcs_enabled = true aft_management_account_id = "REMOVED" aft_metrics_reporting = false aft_vpc_endpoints = false audit_account_id = "REMOVED" ct_home_region = "us-east-1" ct_management_account_id = "REMOVED" log_archive_account_id = "REMOVED" terraform_distribution = "oss" terraform_version = "1.3.6" tf_backend_secondary_region = "us-west-2" vcs_provider = "codecommit" } terraform { backend "s3" { bucket = "REMOVED" key = "aws-control-tower-aft/terraform.tfstate" region = "us-east-1" } required_providers { aws = { source = "hashicorp/aws" version = "4.66.0" } } } provider "aws" { region = "us-east-1" allowed_account_ids = ["REMOVED"] # management account default_tags { tags = { CostCenter = "AFT" } } } ```

  6. Invoked aft-invoke-customizations step function on a single account, completes successfully

  7. Upgrade AFT 1.10.4 --> 1.11.1, with provider 4.66.0 --> 5.11.0 and terraform_version = 1.5.7 in main.tf (BUT using 1.3.6 locally to deploy the module)

    Terraform HCL for AFT @ 1.10.4 ```hcl module "control_tower_account_factory" { source = "aws-ia/control_tower_account_factory/aws" version = "1.11.1" aft_feature_cloudtrail_data_events = false aft_feature_delete_default_vpcs_enabled = true aft_management_account_id = "REMOVED" aft_metrics_reporting = false aft_vpc_endpoints = false audit_account_id = "REMOVED" ct_home_region = "us-east-1" ct_management_account_id = "REMOVED" log_archive_account_id = "REMOVED" terraform_distribution = "oss" terraform_version = "1.5.7" #note: this is bumped to match new minimum, but still using 1.36 to apply the module locally tf_backend_secondary_region = "us-west-2" vcs_provider = "codecommit" } terraform { backend "s3" { bucket = "REMOVED" key = "aws-control-tower-aft/terraform.tfstate" region = "us-east-1" } required_providers { aws = { source = "hashicorp/aws" version = "5.11.0" } } } provider "aws" { region = "us-east-1" allowed_account_ids = ["REMOVED"] # management account default_tags { tags = { CostCenter = "AFT" } } } ```

  8. Invoked aft-invoke-customizations step function on a single account, completes successfully

  9. Changed my local Terraform version from 1.3.6 --> 1.5.7, to match the value of terraform_version = 1.5.7 in module config.

  10. Execute terraform plan, no changes found.

EDIT: Minor edits for typos

snebhu3 commented 6 months ago

@matthewbarreiro , thank you for reaching out and for providing the details. However, we were unable to reproduce the same behavior as highlighted in the issue. An upgrade of AFT deployment version from 1.9.0 to 1.11.1 was successful, and a subsequent invocation of the aft-invoke-customizations was also successful. It is unclear why you would have seen a different behavior.