Hosted Zone Access Denied When Assuming Role

[nathankodilla]

Terraform Version

Core: 0.12.24 AWS: 2.55.0

Affected Resource(s)

aws_route53_zone

Terraform Configuration Files

provider "aws" {
  allowed_account_ids = ["xxxxxxxxx"]
  assume_role {
    role_arn = "arn:aws:iam::xxxxxxxxxx:role/system/terraform"
  }
  region = "us-east-1"
  version = "~> 2.55.0"
}

resource "aws_route53_zone" "name" {
  name = local.internal_domain

  vpc {
    vpc_id = module.vpc.vpc_id
  }

  tags = {
    Environment = local.env
  }

  lifecycle {
    ignore_changes = [vpc]
  }
}

Output

Error: error getting Route 53 Hosted Zone (/hostedzone/ZZZZZZZZZZZZ): AccessDenied: User: arn:aws:sts::yyyyyyyyyyyy:assumed-role/AWSReservedSSO_AdministratorAccess_6adahdkja/email is not authorized to access this resource status code: 403, request id: uuid

Expected Behavior

Refresh route 53 zone query retries information.

Actual Behavior

Refresh route 53 zone query fails.

Steps to Reproduce

terraform apply

Important Factoids

The assumed role has an administrator role, and assuming the role via cli allows me to query this hosted zone. Only this hosted zone fails. No other aws resources fail to refresh. I have attempted every version down to 2.49.0 with the same results. Additionally, I was previously using a profile name from aws credentials file and when switching to assumed roles is when this issue started.

Talking with aws and looking through cloudtrail, the role arn:aws:iam::xxxxxxxxxx:role/system/terraform has not been assumed. Only my initial role of arn:aws:sts::yyyyyyyyyyyy:assumed-role/AWSReservedSSO_AdministratorAccess_6adahdkja/email was used to make the api call. However, as stated previously all other resources work and assume the role to make the api call.

[nathankodilla]

Additional research yielded that it was instead the resource aws_route53_zone_association throwing this error. Performing a terraform state rm on those resources and then letting them be recreated solved the issue. But something is clearly wrong that this was needed in this case.

[nathankodilla]

Finally found the underlying cause. We had a module that had an empty aws provider in it. When using an environment variable of AWS_PROFILE that was inherited by the module provider. Then when switching to assuming roles inside the main aws provider, the module provider was still using the AWS_PROFILE variable.

What is still curious, is I attempted:

module "main" {
  providers = {
    aws =aws
  }
}

But it would still not work and the module provider would not take on the passed in provider.

[pcolmer]

I'm having a similar problem when trying to import an existing zone from Route 53. I've defined the resource:

resource "aws_route53_zone" "linaro-co" {
  provider = "aws.production-account"
  name     = "linaro.co"
}

and this is the provider:

provider "aws" {
  region = "us-east-1"

  assume_role {
    role_arn     = "arn:aws:iam::XXXXXXXXXXXX:role/EC2_Production_Admin"
    session_name = "Terraform-EC2_Production_Admin"
  }

  alias = "production-account"
}

but attempting to import the zone shows that it is not assuming the specified role_arn but, instead, sticking to the role defined in AWS_PROFILE.

[josjaf]

I think I am running into a similar issue. Has this been resolved?

[mateuszkwiatkowski]

Hello,

We use multiple aws providers instead of assume_role and we're experiencing similar issue:

provider "aws" {
  version    = "~> 3.0"
  region     = "us-east-1"
  access_key = var.STAGING_AWS_ACCESS_KEY_ID
  secret_key = var.STAGING_AWS_SECRET_ACCESS_KEY
  alias      = "staging"
}

provider "aws" {
  version    = "~> 3.0"
  region     = "us-east-1"
  access_key = var.PROD_AWS_ACCESS_KEY_ID
  secret_key = var.PROD_AWS_SECRET_ACCESS_KEY
}

resource "aws_route53_zone" "staging_zone" {
  name     = local.staging_new_domain
  provider = aws.staging
}

Error log:

Error: error creating certificate: error: one or more domains had a problem:
221[*.subdomain.example.com] [*.subdomain.example.com] acme: error presenting token: 2 errors occurred:
222 * route53: AccessDenied: User: arn:aws:iam::ACCOUNT_ID:user/devops-automation is not authorized to access this resource
223 status code: 403, request id: b9ad19f3-0d7a-4a5f-adda-115fd0944574
224 * error encountered while presenting token for DNS challenge: route53: AccessDenied: User: arn:aws:iam::ACCOUNT_ID:user/devops-automation is not authorized to access this resource
225 status code: 403, request id: b9ad19f3-0d7a-4a5f-adda-115fd0944574

ACCOUNT_ID in these errors is ID of default aws provider (with no alias specified)

AWS provider version: v3.25.0 Terraform version: 0.13

[ghomem]

I am seeing the problem with an IAM user's token but not with the root user's token. The IAM user can login and browser route53 on the AWS console, so I am wondering whether this is provider related.

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "2.11.1"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

# configure the providers

provider "digitalocean" {
  token = var.do_token
}

provider "aws" {
  region = "eu-west-2"
  access_key = var.aws_key
  secret_key = var.aws_secret

}

[oqrusk]

Any updates?

[KurtLehnardt]

Admin role getting access denied. Might be the trust policy. The terraform code is taken straight from the docs.

│ Error: error creating certificate: error: one or more domains had a problem:
│ [domain1] [domain2] acme: error presenting token: 2 errors occurred:
│       * route53: AccessDenied: User: arn:aws:sts::1234512345:assumed-role/awsAdminrole/a@b.com is not authorized to access this resource
│       status code: 403, request id: aaa
│       * error encountered while presenting token for DNS challenge: route53: AccessDenied: User: arn:aws:sts::1234512345:assumed-role/awsAdminrole/a@b.com is not authorized to access this resource
│       status code: 403, request id: aaa│
│
│
│
│   with module.mydir.acme_certificate.certificate,
│   on modules\mydir\main.tf line 108, in resource "acme_certificate" "certificate":
│  108: resource "acme_certificate" "certificate" {
│
╵

[ieugen]

I'm also seeing this.

UPDATE: For some reason I was missing the "provider = aws.name" for route53 records (dns zone is in another AWS account than ec2 servers). Silly goose me.

When using aws module 4.24 version I get a better error message:

│ Error: AccessDenied: User: arn:aws:sts::XXXX:assumed-role/AWSReservedSSO_AdministratorAccess_d848e235d7407fdc/ is not authorized to access this resource
│       status code: 403, request id: 564c901a-b1c0-4138-b97c-e565f9f73755
│ 
│   with aws_route53_record.XXX[6],
│   on indexing.tf line 169, in resource "aws_route53_record" "XXXXX":
│  169: resource "aws_route53_record" "XXXXX" {

When using aws module version 4.54 .

│ Error: creating Route 53 Record (): getting Hosted Zone (YYYYYY): empty result
│ 
│   with aws_route53_record.XXXX[3],
│   on indexing.tf line 169, in resource "aws_route53_record" "XXXX":
│  169: resource "aws_route53_record" "XXXX" {

The only difference is the aws module version. Config is the same. Terraform state is the same.

The same config used to work some time ago. Probably another terraform version !?

 terraform -version
Terraform v1.3.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v4.54.0
+ provider registry.terraform.io/hashicorp/random v3.3.2

[oblogic7]

I just ran into this same error. I have multiple providers that assume roles and one provider in particular was being used in two instances of a single module. Turns out that duplicating the provider and passing into the second module call prevents the AccessDenied message for my case.

Example code:

provider "aws" {
  alias   = "provider_1"
  profile = "profile_name"
  region  = "us-east-1"

  assume_role {
    role_arn = "arn:aws:iam::1234567890:role/terraform"
  }
}

provider "aws" {
  alias   = "provider_2"
  profile = "profile"
  region  = "us-east-1"

  assume_role {
    role_arn = "arn:aws:iam::1234567890:role/terraform"
  }
}

module "do_something_1" {
  source = "./do_something"

  providers = {
    aws = aws.provider_1
  }
}

module "do_something_2" {
  source = "./do_something"

  providers = {
    aws = aws.provider_2
  }
}