Specifying a profile and role_arn does not work (dynamic role chaining)

joerg commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

$ terraform -v
Terraform v0.14.3
+ provider registry.terraform.io/hashicorp/aws v3.21.0
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.0.0

Affected Resource(s)

aws provider configuration

Terraform Configuration Files

When creating AWS accounts with Terraform I am facing the challenge of "dynamically" creating resources in that new account. According to best practices I am not running Terraform in the organizations account but a "provisioning" account. So my setup is like: provisioning account --role--> organizations account --role--> new account

Creating the account works fine, but then creating resources in the new account from the provisioning account is challenging since I don't have direct access but have to use role chaining via the organizations account.

My approach was to define a profile with the organizations account:

[base]
credential_source = Ec2InstanceMetadata
region            = eu-central-1
role_arn          = arn:aws:iam::<omitted>:role/AccountProvisioning

[organization]
source_profile = base
role_arn       = arn:aws:iam::<omitted>:role/AccountProvisioning
external_id    = <omitted>

And from there assuming the role in the new account

provider "aws" {
  alias  = "other"
  region = "eu-central-1"

  shared_credentials_file = "${path.module}/creds_role_chaining"
  profile                 = "organization"

  assume_role {
    role_arn    = "arn:aws:iam::${var.id}:role/${var.role}"
  }
}

This results in a "IAM Role cannot be assumed" error.

Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::<omitted>:role/AccountProvisioning) cannot be assumed.

There are a number of possible causes of this - the most common are:
  * The credentials used in order to assume the role are invalid
  * The credentials do not have appropriate permission to assume the role
  * The role ARN is not valid

Error: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

To verify that there are no permissions problems I added the new role to the credentials file manually and ran a plan:

[base]
credential_source = Ec2InstanceMetadata
region            = eu-central-1
role_arn          = arn:aws:iam::<omitted>:role/AccountProvisioning

[organization]
source_profile = base
role_arn       = arn:aws:iam::<omitted>:role/AccountProvisioning
external_id    = <omitted>

[new]
source_profile = organization
role_arn       = arn:aws:iam::<omitted>:role/AccountProvisioning

provider "aws" {
  alias  = "other"
  region = "eu-central-1"

  shared_credentials_file = "${path.module}/creds_role_chaining"
  profile                 = "new"
}

Which allows me to plan and apply. This of course is no solution since the account ID used in the credentials is dynamically created by terraform.

Looking at the numerous tickets open regarding role chaining and credentials I think the best approach would be to provide full chaining within terraform an not rely on outside configuration.

Expected Behavior

It should work...

Actual Behavior

IAM Role (arn:aws:iam:::role/AccountProvisioning) cannot be assumed.

joerg commented 3 years ago

I have been testing this matter in more detail and my assumption was correct as it seems. Given the above cross account role chaining setup and this credentials file:

[default]
region = eu-central-1

[base]
credential_source = Ec2InstanceMetadata
role_arn       = arn:aws:iam::<omitted>:role/AccountProvisioning

[organization]
source_profile = base
role_arn       = arn:aws:iam::<omitted>:role/AccountProvisioning
external_id    = <omitted>

A CLI call to assume the role in the new account works:

$ AWS_PROFILE=organization AWS_SHARED_CREDENTIALS_FILE=creds_role_chaining aws sts assume-role --role-arn arn:aws:iam::<omitted>:role/OrganizationAccountAccessRole --role-session-name temp
{
    "AssumedRoleUser": {
        "AssumedRoleId": "<omitted>:temp", 
        "Arn": "arn:aws:sts::<omitted>:assumed-role/OrganizationAccountAccessRole/temp"
    }, 
    "Credentials": {
      <omitted>
    }
}

While trying the same role chain in Terraform does not work:

provider "aws" {
  alias  = "other"
  region = "eu-central-1"

  shared_credentials_file = "${path.module}/creds_role_chaining"
  profile                 = "organization"

  assume_role {
    role_arn    = "arn:aws:iam::${var.id}:role/${var.role}"
    session_name = "temp"
  }
}

Which gives:

TF_LOG=TRACE terraform plan
<long output omitted>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5: 2020/12/19 15:04:12 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:   <Error>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:     <Type>Sender</Type>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:     <Code>AccessDenied</Code>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:     <Message>User: arn:aws:sts::<omitted>:assumed-role/test_delete_me/i-0b1b56f8694577c23 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<omitted>:role/OrganizationAccountAccessRole</Message>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:   </Error>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:   <RequestId>33009143-1e5e-451f-938b-dd51f3466d55</RequestId>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5: </ErrorResponse>
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5: 2020/12/19 15:04:12 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/AssumeRole failed, attempt 0/25, error AccessDenied: User: arn:aws:sts::<omitted>:assumed-role/test_delete_me/i-0b1b56f8694577c23 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<omitted>:role/OrganizationAccountAccessRole
2020-12-19T15:04:12.368Z [DEBUG] plugin.terraform-provider-aws_v3.22.0_x5:  status code: 403, request id: 33009143-1e5e-451f-938b-dd51f3466d55
<omitted>
Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::<omitted>:role/OrganizationAccountAccessRole) cannot be assumed.

There are a number of possible causes of this - the most common are:
  * The credentials used in order to assume the role are invalid
  * The credentials do not have appropriate permission to assume the role
  * The role ARN is not valid

Error: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The debug log clearly shows that Terraform tries to assume the role in the new account (arn:aws:iam::<< omitted >>:role/OrganizationAccountAccessRole) using the instance profile in the provisioning account (arn:aws:sts::<< omitted >>:assumed-role/test_delete_me/i-0b1b56f8694577c23) and not the provided role chain in the credentials file.

joerg commented 3 years ago

For some weird reason I was able to get the credentials file plus the role in Terraform running, but I had to:

Set export AWS_SDK_LOAD_CONFIG=true
Use the default config file location at ~/.aws/config

To me this behaviour is really dirty and not at all what I expected from the documentation.

lvh commented 3 years ago

@joerg When you say you got it working, to be clear, that's with plaintext creds in ~/.aws/credentials?

(I'm trying to do everything with temporary role creds. Which works fine in regular AWS SDKs, just not Hashicorp's.)

joerg commented 3 years ago

No, that was/is with the credentials file from my last comment. So no clear text credentials but the instance profile and role chaining specified in the config.

lvh commented 3 years ago

Ah, perfect. For future people finding this ticket: I was able to make progress with aws-vault exec --server. That has security implications, because it mimics the IMDS meaning anyone who can access that endpoint can grab credentials. However, the IMDS endpoint is something the underlying Go AWS SDK library supports well (otherwise deploying from infra would break!).

github-actions[bot] commented 9 months ago

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

github-actions[bot] commented 7 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws