Error: operation error STS: AssumeRole (error configuring Terraform AWS Provider)

alexhaycock commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.6 on darwin_arm64

provider registry.terraform.io/hashicorp/archive v2.2.0
provider registry.terraform.io/hashicorp/aws v4.29.0
provider registry.terraform.io/hashicorp/null v3.1.1
provider registry.terraform.io/hashicorp/time v0.8.0

This has been an issue for a few months now with older providers

Affected Resource(s)

provider "aws" assume_role
aws_accessanalyzer_analyzer

Terraform Configuration Files

resource "aws_organizations_delegated_administrator" "iam_access_analyzer" {
  account_id        = aws_organizations_account.audit.id
  service_principal = "access-analyzer.amazonaws.com"
}

provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::222222222222:role/admin-role"
  }
  alias  = "audit-eu-central-1"
  region = "eu-central-1"
}

provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::222222222222:role/admin-role"
  }
  alias  = "audit-ap-northeast-1"
  region = "ap-northeast-1"
}

provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::222222222222:role/admin-role"
  }
  alias  = "audit-ap-southeast-1"
  region = "ap-southeast-1"
}

resource "aws_accessanalyzer_analyzer" "audit-eu-central-1" {
  depends_on = [
    aws_organizations_delegated_administrator.iam_access_analyzer
  ]
  analyzer_name = "AccessAnalyser"
  type          = "ORGANIZATION"
  provider      = aws.audit-eu-central-1
}

resource "aws_accessanalyzer_analyzer" "audit-ap-northeast-1" {
  depends_on = [
    aws_organizations_delegated_administrator.iam_access_analyzer
  ]
  analyzer_name = "AccessAnalyser"
  type          = "ORGANIZATION"
  provider      = aws.audit-ap-northeast-1
}

resource "aws_accessanalyzer_analyzer" "audit-ap-southeast-1" {
  depends_on = [
    aws_organizations_delegated_administrator.iam_access_analyzer
  ]
  analyzer_name = "AccessAnalyser"
  type          = "ORGANIZATION"
  provider      = aws.audit-ap-southeast-1
}

Debug Output

Panic Output

Expected Behavior

Deployed access analyzer resource in to all regions

Actual Behavior

│ Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::222222222222:role/admin-role) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 1991a839-35eb-49b4-922f-f5f12cd880c1, api error AccessDenied: User: arn:aws:iam::111111111111:user/admin-role is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/admin-role
│
│
│   with module.core.provider["registry.terraform.io/hashicorp/aws"].audit-eu-central-1,
│   on ../../ecp/ecp-lz-core/provider.tf line 218, in provider "aws":
│  218: provider "aws" {
│
╵
╷
│ Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::222222222222:role/admin-role) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 4e55e93c-97c0-4dd4-ad30-ea95222ca47a, api error AccessDenied: User: arn:aws:iam::111111111111:user/admin-role is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/admin-role
│
│
│   with module.core.provider["registry.terraform.io/hashicorp/aws"].audit-ap-northeast-1,
│   on ../../ecp/ecp-lz-core/provider.tf line 266, in provider "aws":
│  266: provider "aws" {
│
╵
╷
│ Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::222222222222:role/admin-role) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 8f840c28-7107-4740-a50b-4ccebe9e80b5, api error AccessDenied: User: arn:aws:iam::111111111111:user/admin-role is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/admin-role
│
│
│   with module.core.provider["registry.terraform.io/hashicorp/aws"].audit-ap-southeast-1,
│   on ../../ecp/ecp-lz-core/provider.tf line 290, in provider "aws":
│  290: provider "aws" {

Steps to Reproduce

We get this error on the first apply within a new environment, on the second apply it will work as expected.

terraform apply

Important Factoids

We create this resource in all regions that are enabled by default, we don't always get the error can sometimes have it apply with no issues at all. Then other times we get the error in a few regions more often than not it is the ap-* regions where it can't assume the role.

References

rymancl commented 1 year ago

@alexhaycock - Did you ever figure out any cause of this? We're running into this error with the ap-southeast-1 region. The project also has providers for Oregon, N. Virginia and Ireland but Singapore is the only provider where it errors.

I read on a blog post that this error will happen when STS is not enabled for a certain region, but I confirmed all the regions listed above have it enabled.

Unsure if related, but we only started seeing this after trying to migrate from provider v3 (v3.75.2) to v4 (v4.32.0). It seems to be fine with v3.

alexhaycock commented 1 year ago

@rymancl - Nope not got anywhere with it but seeing the same as you, it always seems to happen with a couple of the ap regions. The other regions are hit and miss if it happens or not.

And the same we can see everything is enabled, wondered if it was a timing issue at one point but doesn't seem to matter how long the accounts have been up and running it still happens.

Unfortunately I'm unsure on when or what version we started seeing this issue as we didn't set specific versions but would say around 6 months ago for us we started getting this error. The second apply works all the time but it's annoying that it throws these errors up.

rymancl commented 1 year ago

@alexhaycock - We were able to work around this by adding sts_region to the ap-southeast-1 provider configuration.

provider "aws" {
  alias      = "Singapore"
  region     = "ap-southeast-1"
  sts_region = "us-west-2"
  assume_role {
    role_arn = "<role-arn-here>"
  } 
  # remaining config omitted
}

I don't know if this actually fixed it, but we were able to plan and apply after adding this. I'd be curious if this helps your case.

alexhaycock commented 1 year ago

@rymancl - Just tried your recommendation today and looks like it has fixed it, we got an apply with no errors. Will try this again in the next week to double check it has fixed it.

Thanks a lot for the suggestion!

priyanshur-curefit commented 3 months ago

@alexhaycock - We were able to work around this by adding sts_region to the ap-southeast-1 provider configuration.
provider "aws" {
  alias      = "Singapore"
  region     = "ap-southeast-1"
  sts_region = "us-west-2"
  assume_role {
    role_arn = "<role-arn-here>"
  } 
  # remaining config omitted
}
I don't know if this actually fixed it, but we were able to plan and apply after adding this. I'd be curious if this helps your case.

Hi, I tried using this but the same is not working for "ap-south-2" region. We are getting assume role error only on this region. Any other workaround?

hashicorp / terraform-provider-aws