Provider assume_role fails when using FIPS endpoint for STS and regions other than us-east-1

lorengordon commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

❯ terraform version
Terraform v1.0.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.57.0

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  region = "us-west-2"

  assume_role {
    role_arn = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
  }

  endpoints {
    sts = "https://sts-fips.us-west-2.amazonaws.com"
  }
}

data "aws_caller_identity" "this" {}

Debug Output

2021-09-07T11:26:13.396-0700 [INFO]  provider.terraform-provider-aws_v3.57.0_x5: 2021/09/07 11:26:13 [DEBUG] [aws-sdk-go] DEBUG: Response sts/AssumeRole Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403
Connection: close
Content-Length: 317
Content-Type: text/xml
Date: Tue, 07 Sep 2021 18:26:14 GMT
Keep-Alive: timeout=5
X-Amzn-Requestid: 0f65b116-e91f-4515-9ec8-01b1371f788b

-----------------------------------------------------: timestamp=2021-09-07T11:26:13.396-0700
2021-09-07T11:26:13.396-0700 [INFO]  provider.terraform-provider-aws_v3.57.0_x5: 2021/09/07 11:26:13 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>
  </Error>
  <RequestId>0f65b116-e91f-4515-9ec8-01b1371f788b</RequestId>
</ErrorResponse>: timestamp=2021-09-07T11:26:13.396-0700
2021-09-07T11:26:13.396-0700 [INFO]  provider.terraform-provider-aws_v3.57.0_x5: 2021/09/07 11:26:13 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/AssumeRole failed, attempt 0/25, error SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-east-1'.
        status code: 403, request id: 0f65b116-e91f-4515-9ec8-01b1371f788b: timestamp=2021-09-07T11:26:13.396-0700
2021-09-07T11:26:13.397-0700 [INFO]  backend/local: plan operation completed
╷
│ Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│       For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│
│
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 1, in provider "aws":
│    1: provider "aws" {
│
╵

Expected Behavior

Expected Terraform to use the region from the provider config or the AWS_DEFAULT_REGION environment when performing the assume-role call.

Actual Behavior

Terraform used the us-east-1 region, which failed because the FIPS endpoint is scoped to the region.

Steps to Reproduce

terraform plan

lorengordon commented 3 years ago

Looking at the request in the debug, appears that terraform is not passing the region at all, which is why it is defaulting to us-east-1:

---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts-fips.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.40.35 (go1.16; linux; amd64)
Content-Length: 159
Authorization: ...
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20210907T182613Z
Accept-Encoding: gzip

Action=AssumeRole&DurationSeconds=900&RoleArn=arn%3Aaws%3Aiam%3A%3AACCOUNT_ID%3Arole%2FROLE_NAME&RoleSessionName=1631039173085100400&Version=2011-06-15

Proving the cli works when passing the region:

❯ aws sts assume-role --role-arn arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME --role-session-name SESSION_NAME --endpoint-url https://sts-fips.us-west-2.amazonaws.com --region us-west-2
{
    "Credentials": {
        "AccessKeyId": "...",
        "SecretAccessKey": "...",
        "SessionToken": "...",
        "Expiration": "2021-09-07T19:34:46Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "...",
        "Arn": "arn:aws:sts::ACCOUNT_ID:assumed-role/ROLE_NAME/SESSION_NAME"
    }
}

YakDriver commented 2 years ago

This is a bloody interesting problem. There are two distinct routes for getting creds and environment variables and creds based on environment variables. The testing harness uses its own whole process while the non-testing provider doesn't do a whole lot, letting AWS handle things like environment variables. I'm assuming you are not seeing this error in the testing harness so that eliminates one path.

I see this as two problems since it should work with either way. That neither seems to work is concerning.

If the provider has region set, it absolutely should be passed along. That should override whatever else including the environment variable. There're a few places where "global" services have their regions coerced to their home regions. STS is not one of those. Route53 and Shield are global and have home regions of us-east-1.
This should work fine also if AWS_DEFAULT_REGION is set since AWS will pick that up. (The testing cred path looks at envvars but normal does not.)

lorengordon commented 2 years ago

I'm assuming you are not seeing this error in the testing harness so that eliminates one path.

Well, I'm not running the testing harness, so, no.... 😁

YakDriver commented 2 years ago

The good news, sort of, is that I can reproduce in the harness so.... hmmm...

YakDriver commented 2 years ago

Unfortunately, this cannot be resolved in the provider itself and requires fixes at https://github.com/hashicorp/aws-sdk-go-base. I'll leave this issue open to track the provider-side changes but the base and provider fixes will not be resolved until v4.0.

lorengordon commented 2 years ago

I saw the fips endpoint feature patches in aws-sdk-go-base... But v4!?! That stuff is held up on a major version change? What's the release date on that?

YakDriver commented 2 years ago

What's the release date on that?

ASAP

github-actions[bot] commented 2 years ago

This functionality has been released in v4.0.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws