Cannot execute due to missing provider configuration

smokentar commented 2 years ago

Terraform Version & Prov: 1.2.8

AFT Version: 1.2.1

Bug Description I deployed AFT framework about 7 months ago with GitHub as VCS + TFC as backend. Due to being tied with development I have missed to maintain and regularly update the pipeline.

I am now trying to upgrade from 1.2.1 to 1.6.2 however encountering the following error in TFC when running the aws-control-tower-aft workspace responsible for provisioning infrastructure:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found
with provider["registry.terraform.io/hashicorp/aws"]
on <empty> line 0:

Error: Invalid provider configuration
Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider documentation.

My current configuration is standard:

The variables referenced here are defined in terraform.auto.tfvars.

To Reproduce

Either push an update to GitHub repo to trigger TFC run or manually trigger TFC run from UI
See error

Expected behavior Terraform executes without complaining that a provider configuration cannot be found.

Additional context From the log I can see TFC is failing to find a provider block due to: on <empty> line 0: I can observe these exist in the official terraform-aws-control_tower_account_factory which I am calling, however cannot figure out why I am receiving this error.

Possible RC Upon inspecting providers.tf I can observe they all have aliases. This would indicate there isn't a default aws provider block for resources to fallback to.

From Terraform's provider documentation I have extracted:

A provider block without an alias argument is the default configuration for that provider. Resources that don't set the provider meta-argument will use the default provider configuration

By reviewing the modules consumed by main.tf I can spot a resource that doesn't have a provider set: data "aws_partition" "current" {} This is part of the aft-iam-roles module to which only aliased providers get passed in.

Could this be causing the problem since it will be trying to fallback to a non-existent default aws provider block hence the error I have received?

I can spot aws_partition with no provider on a couple more places: modules/aft-feature-options/data.tf modules/aft-iam-roles/admin-role/data.tf /data.tf modules/aft-code-repositories/data.tf - unaliased aws block passed in so should be okay modules/aft-lambda-layer/data.tf - unaliased aws block passed in so should be okay modules/aft-account-provisioning-framework/data.tf - unaliased aws block passed in so should be okay modules/aft-customizations/data.tf - unaliased aws block passed in so should be okay modules/aft-account-request-framework/data.tf - unaliased aws block passed in so should be okay

snebhu3 commented 2 years ago

@smokentar thank you for reaching out. Please may you confirm that you followed below steps for updating to latest version of AFT? The steps to update would depend on how AFT is deployed in the respective CT environment:

If the AFT repository was cloned and then deployed:
- Update cloned git repository using git pull
- Deploy the changes by executing terraform apply against the CT management account
If the AFT repository is sourced from Github / Terraform Registry directly as a dependency:
- Update AFT repository using terraform init -upgrade
- Deploy the changes by executing terraform apply against the CT management account

smokentar commented 2 years ago

Hi @snebhu3, thanks for your reply.

I am executing this from Terraform Cloud so I don't really have an option to execute terraform init -upgrade. My state is also stored in Terraform Cloud.

I have however changed the source from GitHub to Terraform Registry and forced the latest version to be picked up:

This still results in the same errors:

Terraform v1.2.8
on linux_amd64
Initializing plugins and modules...
module.aft.module.packaging.data.archive_file.builder: Reading...
module.aft.module.packaging.data.archive_file.customizations: Reading...
module.aft.module.packaging.data.archive_file.request_framework: Reading...
module.aft.module.packaging.data.archive_file.feature_options: Reading...
module.aft.module.packaging.data.archive_file.provisioning_framework: Reading...
module.aft.module.aft_customizations.data.local_file.aft_global_customizations_terraform: Reading...
module.aft.module.aft_customizations.data.local_file.aft_account_customizations_terraform: Reading...
module.aft.module.aft_customizations.data.local_file.aft_global_customizations_terraform: Read complete after 0s [id=4566919c984ca209f8b11c3949cde056b29b020d]
module.aft.module.aft_customizations.data.local_file.aft_create_pipeline: Reading...
module.aft.module.aft_lambda_layer.data.local_file.aft_lambda_layer: Reading...
module.aft.module.aft_customizations.data.local_file.aft_create_pipeline: Read complete after 0s [id=278c0e859be0fbe78ebb7dd73f159ef8b3a324be]
module.aft.module.packaging.data.archive_file.builder: Read complete after 0s [id=4c83442cf33de4d3aa36cab6fb65a118c94a3fd7]
module.aft.module.aft_customizations.data.local_file.aft_account_customizations_terraform: Read complete after 0s [id=06584eeb5e044283f51cc7decd799dac27ed3343]
module.aft.module.packaging.data.archive_file.customizations: Read complete after 0s [id=2119ded81635e101a404170ac77e697674cd6e05]
module.aft.module.packaging.data.archive_file.feature_options: Read complete after 0s [id=b104ca9190aa7b99ab73660fd8a300403903a525]
module.aft.module.aft_lambda_layer.random_string.resource_suffix: Refreshing state... [id=kqa80mtw]
module.aft.module.aft_code_repositories.data.local_file.account_request_buildspec: Reading...
module.aft.data.local_file.version: Reading...
module.aft.module.aft_lambda_layer.data.local_file.aft_lambda_layer: Read complete after 0s [id=e7cbe092e97f9311cd3330a8c7e2dbe5dbb13ea9]
module.aft.module.aft_code_repositories.data.local_file.account_provisioning_customizations_buildspec: Reading...
module.aft.module.packaging.data.archive_file.request_framework: Read complete after 0s [id=f57ec97ef8c5e2dedee631903e6eddfd40c33bf8]
module.aft.module.aft_code_repositories.data.local_file.account_request_buildspec: Read complete after 0s [id=f9249e8cc0c976cf31c472ddc9bdb4f628656964]
module.aft.module.aft_code_repositories.data.local_file.account_provisioning_customizations_buildspec: Read complete after 0s [id=d90be7f394cc43afe8df7d0e36ebfe7aafa1f878]
module.aft.data.local_file.version: Read complete after 0s [id=023695a7542e76cdbe9c79cc349ed4a5f5325749]
module.aft.module.packaging.data.archive_file.provisioning_framework: Read complete after 0s [id=7e4a7ac3a7772ea7132b0982c26aeca7dae175fe]
module.aft.module.aft_lambda_layer.time_sleep.eventbridge_rule: Refreshing state... [id=2022-02-01T20:30:15Z]
module.aft.module.aft_lambda_layer.time_sleep.lambda_layer_wait: Refreshing state... [id=2022-02-01T20:37:17Z]
module.aft.module.aft_account_request_framework.data.aws_caller_identity.ct-management: Reading...
module.aft.module.aft_account_request_framework.aws_cloudwatch_event_rule.aft_control_tower_events: Refreshing state... [id=aft-capture-ct-events]
module.aft.module.aft_feature_options.data.aws_caller_identity.ct_management: Reading...
module.aft.module.aft_iam_roles.module.ct_management_exec_role.data.aws_partition.current: Reading...
module.aft.module.aft_feature_options.data.aws_organizations_organization.aft_organization: Reading...
module.aft.module.aft_account_request_framework.aws_iam_role.aft_control_tower_events: Refreshing state... [id=aft-control-tower-events-rule]
module.aft.module.aft_iam_roles.module.ct_management_exec_role.data.aws_partition.current: Read complete after 0s [id=aws]
module.aft.module.aft_feature_options.data.aws_caller_identity.ct_management: Read complete after 0s [id=243542532321]
module.aft.module.aft_account_request_framework.data.aws_caller_identity.ct-management: Read complete after 0s [id=243542532321]
module.aft.module.aft_feature_options.data.aws_organizations_organization.aft_organization: Read complete after 0s [id=o-okhvlky48k]
╷
│ Error: Invalid provider configuration
│ 
│ Provider "registry.terraform.io/hashicorp/aws" requires explicit
│ configuration. Add a provider block to the root module and configure the
│ provider's required arguments as described in the provider documentation.
│ 
╵
╷
│ Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on <empty> line 0:
│   (source code not available)
│ 
╵
Operation failed: failed running terraform plan (exit 1)

smokentar commented 2 years ago

Hi @snebhu3, after forking the official repository and doing some testing I have confirmed the above RC and pinpointed the issue to the following files:

In both files, data "aws_partition" "current" {} should have a provider specified. Something like:

data "aws_partition" "current" {
  provider = aws.ct_management
}

I am using the ct_management alias as an example.

After adding this I managed to execute a successful plan.

snebhu3 commented 2 years ago

@smokentar thank you for additional context. We tried to reproduce this issue, however, we were unable to. If you need additional help, we would recommend reaching out to AWS Premium Support.

smokentar commented 2 years ago

Hi @snebhu3, thank you for responding. I can confirm the issue is not reproducible when running with Terraform OSS.

Have you attempted to reproduce with Terraform Cloud? It is a supported backend as per the Readme and examples

snebhu3 commented 2 years ago

Hi @smokentar, thank you for your response. We were not able to reproduce this error in both our Terraform Cloud and OSS environments.

smokentar commented 2 years ago

Hi @snebhu3, thanks for following up.

Could you please confirm the AFT version your Terraform Cloud is using is 1.5.2 or above?

I have successfully managed to update my AFT version (plan + apply) to 1.5.1 from 1.2.1. However when I bump to 1.5.2 and try to apply I receive the error mentioned in my first comment. I also receive the same error when trying 1.6.0, 1.6.1 and 1.6.2.

1.5.2 was the version where the aws_partition data source was added. This doesn't make sense to me as we are both using Terraform Cloud to plan and apply - the only difference in behaviour can be from the version used - please confirm the AFT version you used.

Thank you!

snebhu3 commented 2 years ago

Hi @smokentar, thanks for providing more details. We are using latest version of AFT (1.6.2) in Terraform Cloud workspace which uses Terraform version 1.2.8.

smokentar commented 2 years ago

Hi @snebhu3, thanks for confirming!

Could you check if there are any environment variables defined for your TF Cloud workspace? I managed to get a successful plan for 1.5.2 an above by defining an AWS_DEFAULT_REGION environment variable. At this point I'm trying to figure out what's the problem with my TF Cloud configuration.

snebhu3 commented 2 years ago

@smokentar yes, we do use some environment variables in our TF cloud workspace. Since we were not able to reproduce the reported error, we suspect the failure to update to latest AFT could be due to how the workspace / environment is set up. We recommend working with AWS Premium Support or HashiCorp if further help on troubleshooting is required. I will go ahead and close this issue for now.

Please feel free to open another issue if you face any additional problems with AFT.

theipster commented 1 year ago

This issue is reproducible - we've managed to replicate it in two different ways: upgrading from 1.5.1 to 1.5.2 and also upgrading from 1.5.1 to 1.8.0 (latest).

We are using Terraform OSS, but this is actually irrelevant - see below.

The issue is along the same lines as what @smokentar suggested in https://github.com/aws-ia/terraform-aws-control_tower_account_factory/issues/232#issuecomment-1230850364, i.e. there are a few aws_partition data sources that are missing a provider meta-argument.

Here is a suggestion for how to resolve it (we've tested this solution on our local fork): https://github.com/aws-ia/terraform-aws-control_tower_account_factory/pull/303.

It is particularly telling because you can see that similar data sources immediately above and below those lines of code already have a provider = aws.aft_management defined.

Hi @snebhu3, thanks for confirming!

Could you check if there are any environment variables defined for your TF Cloud workspace?

@smokentar yes, we do use some environment variables in our TF cloud workspace.

This is probably the reason why you are unable to replicate the issue. If you have AWS credentials in your environment, then any missing providers will silently fall back to those.

jeremyrp commented 1 year ago

I can confirm that this issue was exhibited in a brand new Control Tower org v3 and accounts. No resources have been provisioned, nor any additional setup done.
TF v1.3.7 AFT 1.8.0 This system has no ~/.aws/config or ~/.aws/credentials. Secrets are defined in env vars. Once I defined the default region (export AWS_DEFAULT_REGION=foo), the error was resolved

aws-ia / terraform-aws-control_tower_account_factory

Cannot execute due to missing provider configuration #232