How to use with HCP Vault?

[SamuelM333]

I'm following the quickstart and following this video AWS re:Invent - Using Vault with AWS Lambda and More with a Vault server hosted on Hashicorp Cloud Platform.

I'm stuck at the third step of the guide, running vault write auth/aws/role/vault-lambda-role

Around the minute 16:12 of the video I linked a similar issue is mentioned, but the solution is not very clear.

These are the commands I ran:

Configure my Vault client

export VAULT_ADDR="<hcp_vault_addr>"
export VAULT_TOKEN="<hcp_vault_token>"
export VAULT_NAMESPACE=admin

Enable AWS auth backend and configure it as the guide says

vault auth enable aws
Success! Enabled aws auth method at: aws/

vault write -force auth/aws/config/client
Success! Data written to: auth/aws/config/client

Add a config STS since my lambda and its role are not in the same account as my user:

vault write auth/aws/config/sts/*** auth_type=iam sts_role="arn:aws:iam::***:role/vault-lambda-role"
Success! Data written to: auth/aws/config/sts/***

Then run the final command from the guide

vault write auth/aws/role/vault-lambda-role \
    auth_type=iam \
    bound_iam_principal_arn="arn:aws:iam::***:role/vault-lambda-role" \
    policies="serverless" \
    ttl=1h
Error writing data to auth/aws/role/vault-lambda-role: Error making API request.

URL: PUT https://***.aws.hashicorp.cloud:8200/v1/auth/aws/role/vault-lambda-role
Code: 400. Errors:

* unable to resolve ARN "arn:aws:iam::***:role/vault-lambda-role" to internal ID: AccessDenied: User: arn:aws:sts::285268573241:assumed-role/HCP-Vault-333b91a7-4002-4181-b026-ca105a6eca86-VaultNode/i-06f09e6fc27076955 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::***:role/vault-lambda-role
    status code: 403, request id: 3044253b-6b33-4f8b-93a5-935c600fd211

This is the assume role policy of my role:

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

I already tried allowing the assumed role from the error message, with no success:

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }

  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type = "AWS"
      identifiers = [
        "arn:aws:sts::285268573241:assumed-role/HCP-Vault-333b91a7-4002-4181-b026-ca105a6eca86-VaultNode/i-06f09e6fc27076955"
      ]
    }
  }
}

Am I missing something? How do I allow the HCP Vault user to assume my lambda role? Any help would be appreciated. Thanks!

[tomhjp]

This is an excellent question. Normally, you would be in control of both Vault's and its client's AWS accounts, however, in this case you only control the client's AWS account. Unfortunately, AFAIK this means you are unable to configure cross-account permissions as would normally be required. The only workaround I know for now is to set resolve_aws_unique_ids=false when creating the role in Vault:

vault write auth/aws/role/vault-lambda-role \
    auth_type=iam \
    bound_iam_principal_arn="arn:aws:iam::***:role/vault-lambda-role" \
    policies="serverless" \
    ttl=1h \
    resolve_aws_unique_ids=false

This does introduce a limitation; the ARN you reference for bound_iam_principal_arn must be in the form you've given it, because the setting instructs Vault not to resolve ARNs into unique IDs and so it can only pattern match between the ARNs given to it. Vault still properly validates the identity of the role and it's still a secure option, just more restrictive in how you can configure it.

This is something that's been flagged internally for HCP Vault, and I can't give any timeframe, but I'm hopeful we'll be able to improve this UX in future.

In the meantime, I'm going to leave this issue open as a documentation issue, as it would be nice to surface this more easily for everyone.

[SamuelM333]

Awesome, that did it! Thanks a lot!

This does introduce a limitation; the ARN you reference for bound_iam_principal_arn must be in the form you've given it

Does that mean that the ARN will be interpreted literally and won't accept wildcards (e.g. arn:aws:iam::***:role/vault*)?

[mustafaezer]

I've been searching for a solution for this exact scenario in our use case for probably the last 4 days. 🤦🏻‍♂️

Thanks for the detailed explanation and solution as well. Looking forward to keeping up with the updates on this one. ✨

[tomhjp]

Thanks for the ping on this. We also now have this learn tutorial, which could readily be adapted for vault-lambda-extension and makes an improvement on my previous comment: https://developer.hashicorp.com/vault/tutorials/cloud-ops/vault-auth-method-aws. I'll leave this issue open though as I'd like to adapt that content specifically for HCP Vault + the Lambda extension in the docs here