tf_backend_secondary_region appears to be required even with terraform cloud

[AliSajid]

Terraform Version & Prov:

AFT Version: N/A

Terraform Version & Provider Versions Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version

Terraform v1.4.6
on darwin_arm64
+ provider registry.terraform.io/auth0/auth0 v0.48.0
+ provider registry.terraform.io/hashicorp/archive v2.3.0
+ provider registry.terraform.io/hashicorp/aws v4.67.0
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/hashicorp/tfe v0.44.1
+ provider registry.terraform.io/hashicorp/time v0.9.1

terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/random] ~> 3.5.1
├── provider[registry.terraform.io/auth0/auth0] ~> 0.48.0
├── provider[registry.terraform.io/hashicorp/tfe] ~> 0.44.0
├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0
└── module.aft
    ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0
    ├── provider[registry.terraform.io/hashicorp/local]
    ├── module.packaging
    │   └── provider[registry.terraform.io/hashicorp/archive]
    ├── module.aft_code_repositories
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/local]
    ├── module.aft_backend
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_feature_options
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_account_provisioning_framework
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0
    ├── module.aft_customizations
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/local]
    ├── module.aft_iam_roles
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.audit_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.ct_management_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.ct_management_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.log_archive_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.log_archive_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.aft_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.aft_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── module.audit_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_account_request_framework
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0
    │   └── provider[registry.terraform.io/hashicorp/time]
    ├── module.aft_lambda_layer
    │   ├── provider[registry.terraform.io/hashicorp/local]
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/random]
    └── module.aft_ssm_parameters
        ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
        └── provider[registry.terraform.io/hashicorp/random]

Bug Description When I try to run the AFT (Step 5 of the deployment steps), I see an error about configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.. On deeper investigation, I could create a different error when I set the tf_backend_secondary_region variable. I am using terraform cloud and have the terraform_distribution parameter set as such. I do not understand why this is happening.

To Reproduce Steps to reproduce the behavior:

1. Create a new module with the shown main.tf
2. Run terraform plan
3. See error

Expected behavior To be able to run it all to completion.

Related Logs

module.aft.module.aft_account_provisioning_framework.data.aws_iam_policy.AWSLambdaBasicExecutionRole: Refresh complete after 0s [id=arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole]
╷
│ Error: configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ AWS Error: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, an AWS region is required, but was not found
│ 
│ 
│   with module.aft.provider["registry.terraform.io/hashicorp/aws"].tf_backend_secondary_region,
│   on .terraform/modules/aft/providers.tf line 28, in provider "aws":
│   28: provider "aws" {
│ 
╵
Operation failed: failed running terraform plan (exit 1)
ERRO[0027] Terraform invocation failed in /Users/aimami/experiments/project_nimbus/Terraform-Core/terraform/TF_account_factory 
ERRO[0027] 1 error occurred:

Additional context Here's my terraform file

module "aft" {
  source  = "aws-ia/control_tower_account_factory/aws"
  version = "1.10.3"
  # Required Vars
  ct_management_account_id    = "xxxxxxxxxxxx"
  log_archive_account_id      = "xxxxxxxxxxxx"
  audit_account_id            = "xxxxxxxxxxxx"
  aft_management_account_id   = "xxxxxxxxxxxx"
  ct_home_region              = "us-east-2"
  # VCS Vars
  vcs_provider                                  = "github"
  account_request_repo_name                     = "AFT-account-request"
  global_customizations_repo_name               = "AFT-global-customizations"
  account_customizations_repo_name              = "AFT-account-customizations"
  account_provisioning_customizations_repo_name = "AFT-account-provisioning-customizations"
  # Terraform Vars
  terraform_org_name = "MyOrg"
  terraform_distribution = "tfc"
}

[balltrev]

Thanks for the report @AliSajid. Was this a fresh deployment of AFT? I wasn't able to reproduce your specific failure case, as I was able to deploy an instance of AFT homed in a single region, using Terraform Cloud.

I have however seen your initial error before.

configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.

There's a potential you need to set an AWS_DEFAULT_REGION for the hashicorp/aws provider to detect. You may try setting that environment variable to see if the provider error clears up.

[AliSajid]

It is a fresh deployment. I do have the default region set in my config. In AWS, I also have other regions turned off.

[snebhu3]

@AliSajid we recommend you to reach out to AWS Premium support to receive further help on troubleshooting the issue in your environment.