Unable to use Recovery KMS HCL Provider Option

[celestialorb]

Terraform Version

v1.1.5

Affected Resource(s)

Hashicorp Boundary Provider

Terraform Configuration Files

provider "boundary" {
  addr = "https://<boundary hostname>"
  recovery_kms_hcl = <<EOT
kms "awskms" {
  purpose = "recovery"
  region = "eu-central-1"
  kms_key_id = "<KEY_ID>"
}
EOT
}

Debug Output

Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated.

Expected Behavior

I would expect the recovery KMS to be able to determine my credentials as I'm using an AWS profile configured via AWS SSO. I have tried explicitly setting the profile in the kms stanza as well as explicitly setting an access key id and secret access key. All return the same error. Every set of credentials I've tried all have full admin access as well, though the error message suggests that it can't find the credentials rather than the credentials don't have the proper authorization to read AWS KMS.

Actual Behavior

The error above always appears. Configuring credentials for the stanza doesn't seem to work.

Additional Information

Maybe #180 would help?

[malnick]

Sorry for the tardy response here @celestialorb - can you retry with the latest provider build? We've updated the go-kms lib within the provider.

[celestialorb]

Went through the motions with the latest provider. Some progress has been made in that I can now get the provider to work with the recovery_kms_hcl attribute if I explicitly set the access key, secret key, and session token in the stanza (or via the environment). However I still cannot get it to work by specifying a profile.

Seems like it has issues if a profile is configured to obtain its credentials via AWS SSO. If I add a new profile on my machine with the access key, secret key, and session token explicitly defined the provider works but using a profile configured to use AWS SSO does not and fails with the same error message originally mentioned.

Has this been tested with a profile the utilizes AWS SSO to obtain the credentials?

[malnick]

Not that I'm aware of, the KMS recovery method uses our KMS wrapping library, which relies on the AWS SDK at the end of the day for credential chains. Reading the docs on AWS SSO, it sounds like you should be prompted for a browser to login through to get your credentials. Do you get that prompt? Can you share the error you receive?

FYI @jimlambrt @jefferai Something we may want to look at upstream in the KMS wrapping library.

[jefferai]

The KMS wrapping library uses https://github.com/hashicorp/go-secure-stdlib/tree/main/awsutil for creds, it's not something in the wrapper itself.

[celestialorb]

@malnick I should only be prompted when I perform an SSO login (aws sso login) which I usually do in the mornings and the credentials from that last for something like 8 or 12 hours. When running a terraform plan with the Boundary recovery-configured provider I receive a NoCredentialProviders error.

╷ │ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated. │ For verbose messaging see aws.Config.CredentialsChainVerboseErrors │ │ with provider["registry.terraform.io/hashicorp/boundary"].recovery, │ on main.tf line 56, in provider "boundary": │ 56: provider "boundary" { │ ╵

I am able to authenticate locally in my shell, which was verified by running aws sts get-caller-identity, and my Boundary provider's recovery_kms_hcl field resembles the following:

kms "awskms" {
  purpose    = "recovery"
  region     = "eu-central-1"
  kms_key_id = "<AWS KMS key ID here>"
}

I have no other fields specified in the Boundary provider aside from addr.

This certainly feels like an issue in an upstream AWS SDK or the SDK version used is a bit old, either that or perhaps some sort of explicit configuration is being passed to the AWS SDK that effectively makes it skip the SSO option?

[malnick]

Sorry for the very tardy response here but we have an update coming down to move us to v2 of AWS's SDK that may be relevant to this issue: https://github.com/hashicorp/go-secure-stdlib/pull/83

[KrisJohnstone]

Still broken sadly :(

❯ terraform version
Terraform v1.7.5
on darwin_arm64

provider "boundary" {
  addr             = var.url
  recovery_kms_hcl = <<EOT
kms "awskms" {
    purpose    = "recovery"
  kms_key_id = "${var.kms_recovery_key_id}"
}
EOT
}

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error setting configuration on the kms plugin: rpc error: code = Unknown desc = error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated.
│   For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│
│   with module.boundary.provider["registry.terraform.io/hashicorp/boundary"],
│   on boundary/main.tf line 13, in provider "boundary":
│   13: provider "boundary" {
│
╵

[KrisJohnstone]

Created a boundary sso workaround user:

With the following policy attached:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:Decrypt",
                "kms:Encrypt"
            ],
            "Resource": [
                "arn:aws:kms:<REGION>:<ACCOUNTID>:key/<KEYID>"
            ]
        }
    ]
}

Modified the Provider:

provider "boundary" {
  addr             = var.url
  recovery_kms_hcl = <<EOT
kms "awskms" {
    purpose    = "recovery"
  kms_key_id = "${var.kms_recovery_key_id}"
  access_key = "your accesskey"
  secret_key = "your secretkey"
}
EOT
}

Which got me passed the error.