Unable to assume role with a specified STS endpoint

ghost commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.29 provider.aws v3.0.0

Affected Resource(s)

aws (Provider)

Terraform Configuration Files

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::XXXXXXXXXXXX:role/test"  # Role with source VPC condition
  }
  endpoints {
    sts = "https://sts.eu-west-1.amazonaws.com"
  }
}

resource "aws_iam_user" "test" {
  name = "test"
}

Debug Output

https://gist.github.com/xlz-jgoutin/f2d8e56f8ff179e8dc67dced38f15196

Panic Output

Expected Behavior

Assume role with success

Actual Behavior

Assume role fail due to VPC condition (Like when the endpoint is not specified).

Steps to Reproduce

terraform apply

Important Factoids

The role has a condition that allow only to be assumed from a specific VPC in eu-west-1 region
The VPC in eu-west-1 region has a STS VPC endpoint.
No issue when assuming the role using awscli with the "--endpoint-url https://sts.eu-west-1.amazonaws.com" argument.
No issue when using "AWS_STS_REGIONAL_ENDPOINTS" (But this does not allow to force an endpoint in a different region than the managed one)

References

14077: From changelog "provider: Ensure configured STS endpoint is used during AssumeRole API calls"

ewbankkit commented 4 years ago

Danny-Cooke commented 4 years ago

Is there a planned date to deliver a fix for this issue? We are bound by security to use VPC endpoints and so this causes us many problems.

Just to add further detail with Debug enabled the terraform run just hangs on the line "Setting custom STS endpoint"

ewbankkit commented 4 years ago

@Danny-Cooke Are you using a similar cross-region setup where the region specified in the provider block is not the region for the STS endpoint? For example:

provider "aws" {
  region = "us-west-2"

  sts_endpoint = "https://sts.eu-central-1.amazonaws.com"
}

Danny-Cooke commented 4 years ago

Actually no we are not using cross region but we are using cross account roles.

this is the layout of our provider file.and we are trying to have a "shared services" account which can provision into all other accounts. I have left in (but commented out) our other attemps to use regional endpoints for STS too but alas, we have no success with it.

provider "aws" { version = "~> 3.9.0" region = "eu-west-1"

endpoints { s3 = "https://s3.eu-west-1.amazonaws.com"

ssm                           = "vpce-00000000000000000.ssm.eu-west-1.vpce.amazonaws.com"
ec2                            = "vpce-00000000000000000.eu-west-1.vpce.amazonaws.com"
secretsmanager        = "vpce-00000000000000000.secretsmanager.eu-west-1.vpce.amazonaws.com"
sts                             = "vpce-00000000000000000.sts.eu-west-1.vpce.amazonaws.com"
##sts                         = "sts.eu-west-1.vpce.amazonaws.com"
##sts                         = "sts.eu-west-1.amazonaws.com"

} assume_role { role_arn = "arn:aws:iam::111111111111:role/operational_role" } }

ewbankkit commented 4 years ago

@Danny-Cooke Yes, I can reproduce this with a same-region scenario:

provider "aws" {
  region = "us-west-2"

  assume_role {
    role_arn = "arn:aws:iam::111111111111:role/test"
  }

  endpoints {
    sts = "https://sts.us-west-2.amazonaws.com"
  }
}

2020-10-06T09:10:21.419-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [INFO] assume_role configuration set: (ARN: "arn:aws:iam::111111111111:role/test", SessionID: "", ExternalID: "")
2020-10-06T09:10:21.427-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [INFO] AWS Auth provider used: "EnvProvider"
2020-10-06T09:10:21.427-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [INFO] Attempting to AssumeRole arn:aws:iam::111111111111:role/test (SessionName: "", ExternalId: "")
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [INFO] Setting custom STS endpoint: https://sts.us-west-2.amazonaws.com
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [DEBUG] [aws-sdk-go] DEBUG: Request sts/AssumeRole Details:
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: ---[ REQUEST POST-SIGN ]-----------------------------
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: POST / HTTP/1.1
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Host: sts.us-west-2.amazonaws.com
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: User-Agent: aws-sdk-go/1.35.2 (go1.14.6; linux; amd64)
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Content-Length: 153
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20201006/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=4cc9086b28c9e83f6681a503c6b4c69fcea1c7a302770fc542130a0fe40339e0
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: X-Amz-Date: 20201006T131021Z
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Accept-Encoding: gzip
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: 
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: Action=AssumeRole&DurationSeconds=900&RoleArn=arn%3Aaws%3Aiam%3A%3A111111111111%3Arole%2Ftest&RoleSessionName=1601989821428130890&Version=2011-06-15
2020-10-06T09:10:21.428-0400 [DEBUG] plugin.terraform-provider-aws: -----------------------------------------------------
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [DEBUG] [aws-sdk-go] DEBUG: Response sts/AssumeRole Details:
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: ---[ RESPONSE ]--------------------------------------
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: HTTP/1.1 403 Forbidden
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: Connection: close
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: Content-Length: 317
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: Content-Type: text/xml
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: Date: Tue, 06 Oct 2020 13:10:21 GMT
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: X-Amzn-Requestid: 3534c331-31aa-4227-829c-fc887c1b0b2e
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: 
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: 
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: -----------------------------------------------------
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2020-10-06T09:10:21.919-0400 [DEBUG] plugin.terraform-provider-aws:   <Error>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:     <Type>Sender</Type>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:     <Code>SignatureDoesNotMatch</Code>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:     <Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:   </Error>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:   <RequestId>3534c331-31aa-4227-829c-fc887c1b0b2e</RequestId>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws: </ErrorResponse>
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 09:10:21 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/AssumeRole failed, attempt 0/25, error SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-east-1'. 
2020-10-06T09:10:21.920-0400 [DEBUG] plugin.terraform-provider-aws:     status code: 403, request id: 3534c331-31aa-4227-829c-fc887c1b0b2e
2020/10/06 09:10:21 [ERROR] <root>: eval: *terraform.EvalConfigProvider, err: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::111111111111:role/test) cannot be assumed.

which is the error called out in #14873.

If I make the equivalent call using the CLI

$ aws --debug --region us-west-2 --endpoint https://sts.us-west-2.amazonaws.com sts assume-role --role-arn arn:aws:iam::111111111111:role/test --role-session-name testing

it succeeds:

2020-10-06 09:23:32,370 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.us-west-2.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.18.142 Python/3.6.9 Linux/5.4.0-48-generic botocore/1.18.1', 'X-Amz-Date': b'20201006T132332Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20201006/us-west-2/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=d247a9d62006c1345877163392ba149f57ce1cb0591d719bb6c16499a9382012', 'Content-Length': '121'}>
2020-10-06 09:23:32,371 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.us-west-2.amazonaws.com:443
2020-10-06 09:23:32,935 - MainThread - urllib3.connectionpool - DEBUG - https://sts.us-west-2.amazonaws.com:443 "POST / HTTP/1.1" 200 1438
2020-10-06 09:23:32,937 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '876945ab-53ae-4cb6-b264-f1a11938039e', 'Content-Type': 'text/xml', 'Content-Length': '1438', 'Date': 'Tue, 06 Oct 2020 13:23:32 GMT'}

The difference is that the credential signing region (Credential= in the Authorization HTTP header) is us-east-1 from the AWS Provider and us-west-2 from the CLI.

ewbankkit commented 4 years ago

@Danny-Cooke Could you try setting the AWS_STS_REGIONAL_ENDPOINTS environment variable to the value regional? I'm no expert at this but that MAY force the signing region to be set to the correct region in this case.

Danny-Cooke commented 4 years ago

@ewaltman We did try that but with no positive impact.

we also use Aviatrix Filters and whitelisting to access the internet where endpoints are not available. When trying to hit the VPC endpoint weather its with AWS_STS_REGIONAL_ENDPOINTS or not we check the filter to find terraform has still tried to access the same global address of "sts.amazonaws.com" every time.

It seems like regardless, terraform still wants to go external. For the time being we are having to whitelist that address until a fix is delivered?

ewbankkit commented 4 years ago

@xlz-jgoutin

No issue when assuming the role using awscli with the "--endpoint-url https://sts.eu-west-1.amazonaws.com" argument.

Which region are you specifying or have set for the successful sts assume-role call?

ewbankkit commented 4 years ago

Hmm, when I set AWS_STS_REGIONAL_ENDPOINTS to regional and don't specify an STS endpoint in my configuration

provider "aws" {
  region = "us-west-2"

  assume_role {
    role_arn = "arn:aws:iam::111111111111:role/test"
  }

  # endpoints {
  #   sts = "https://sts.us-west-2.amazonaws.com"
  # }
}

then

$ AWS_STS_REGIONAL_ENDPOINTS=regional terraform plan

succeeds via the correct regional endpoint:

2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 11:58:56 [DEBUG] [aws-sdk-go] DEBUG: Request sts/AssumeRole Details:
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: ---[ REQUEST POST-SIGN ]-----------------------------
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: POST / HTTP/1.1
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Host: sts.us-west-2.amazonaws.com
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: User-Agent: aws-sdk-go/1.35.2 (go1.14.6; linux; amd64)
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Content-Length: 153
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXX/20201006/us-west-2/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=120f7a596e3e26df7e83713c0964f7d194842875d594b89d7e4328fd70c3d27d
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: X-Amz-Date: 20201006T155856Z
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Accept-Encoding: gzip
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: 
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: Action=AssumeRole&DurationSeconds=900&RoleArn=arn%3Aaws%3Aiam%3A%111111111111%3Arole%2Ftest001.1&RoleSessionName=1601999936413470788&Version=2011-06-15
2020-10-06T11:58:56.413-0400 [DEBUG] plugin.terraform-provider-aws: -----------------------------------------------------
2020-10-06T11:58:56.979-0400 [DEBUG] plugin.terraform-provider-aws: 2020/10/06 11:58:56 [DEBUG] [aws-sdk-go] DEBUG: Response sts/AssumeRole Details:
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: ---[ RESPONSE ]--------------------------------------
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: HTTP/1.1 200 OK
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: Connection: close
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: Content-Length: 1478
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: Content-Type: text/xml
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: Date: Tue, 06 Oct 2020 15:58:56 GMT
2020-10-06T11:58:56.980-0400 [DEBUG] plugin.terraform-provider-aws: X-Amzn-Requestid: d3851423-1fd0-4838-8833-b51093bbea63

Danny-Cooke commented 4 years ago

@ewaltman That's interesting. we haven't tried the combination of NOT specifying the sts endpoint. Basically as you should be specifying them all the time. I'll get the network guys to remove the whitelist tomorrow and we will test too.

Definitely some strange results going on

Danny-Cooke commented 4 years ago

@ewaltman i can confirm the same results. STS endpoint is created STS endpoint not specified in provider

provider "aws" { version = "~> 3.9.0" region = "eu-west-1"

endpoints { s3 = "https://s3.eu-west-1.amazonaws.com" ssm = "vpce-0000000000000000-uhkbpo62.ssm.eu-west-1.vpce.amazonaws.com" ec2 = "vpce-0000000000000000-j5jsaeba.ec2.eu-west-1.vpce.amazonaws.com" secretsmanager = "vpce-0000000000000000-xe7lzrhd.secretsmanager.eu-west-1.vpce.amazonaws.com" } }

export AWS_STS_REGIONAL_ENDPOINTS=regional terraform apply

we can then successfuly assume role into another account and deploy

ewbankkit commented 4 years ago

Would an attribute something like

provider "aws" {
  region = "us-west-2"

  assume_role {
    role_arn = "arn:aws:iam::111111111111:role/test"

    sts_regional_endpoints = "regional"
  }
}

be of use so that the AWS_STS_REGIONAL_ENDPOINTS environment variable does not nee to be set?

The cross-region case is more complex and will require specifying a signing region for the STS AssumeRole request (else the signing region is either us-east-1 or the provider's region if AWS_STS_REGIONAL_ENDPOINTS=regional).

ghost commented 3 years ago

@xlz-jgoutin

No issue when assuming the role using awscli with the "--endpoint-url https://sts.eu-west-1.amazonaws.com" argument.

Which region are you specifying or have set for the successful sts assume-role call?

us-east-1.

msahihi commented 3 years ago

Same issue here

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.28 provider.aws v3.38

2021/05/12 09:02:17 [TRACE] GRPCProvider: Configure
2021-05-12T09:02:17.925Z [INFO]  plugin.terraform-provider-aws_v3.38.0_x5: 2021/05/12 09:02:17 [INFO] AWS Auth provider used: "StaticProvider": timestamp=2021-05-12T09:02:17.925Z
2021-05-12T09:02:17.926Z [INFO]  plugin.terraform-provider-aws_v3.38.0_x5: 2021/05/12 09:02:17 [INFO] Setting custom STS endpoint: https://sts.eu-central-1.amazonaws.com: timestamp=2021-05-12T09:02:17.925Z
2021-05-12T09:02:17.926Z [INFO]  plugin.terraform-provider-aws_v3.38.0_x5: 2021/05/12 09:02:17 [DEBUG] Trying to get account information via sts:GetCallerIdentity: timestamp=2021-05-12T09:02:17.925Z
2021-05-12T09:02:17.929Z [INFO]  plugin.terraform-provider-aws_v3.38.0_x5: 2021/05/12 09:02:17 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.eu-central-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/0.12.28 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.38.29 (go1.16; linux; amd64)
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXX/20210512/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=edfd3d2b4412e10307c92ecf0b01d431c6bea594f7422ecf4b314b3ab4cb88b6
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20210512T090217Z
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------: timestamp=2021-05-12T09:02:17.927Z
2021-05-12T09:02:18.655Z [INFO]  plugin.terraform-provider-aws_v3.38.0_x5: 2021/05/12 09:02:18 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Content-Length: 317
Content-Type: text/xml
Date: Wed, 12 May 2021 09:02:17 GMT
X-Amzn-Requestid: 67ae2cc5-5498-4c43-b24f-358637069498

rkitron commented 3 years ago

Hi,

I have the same needs.

Based on the documentation about Global STS and Regional STS, the provider should do the action in two-step:

If sts endpoints (a regional one), meaning sts..amazonaws.com, the provider needs to do the sts:assumerole in that region regardless of the "region" parameter. Then as the credentials can now be used in whatever region, the provider should now switch to that region.

Benoit,

ewbankkit commented 2 years ago

See https://github.com/boto/botocore/blob/04d1fae43b657952e49b21d16daa86378ddb4253/botocore/args.py#L84-L93 for an example of signing region.

github-actions[bot] commented 5 months ago

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

github-actions[bot] commented 2 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws