Open kirkdave opened 3 years ago
I got this to work by manually adding the AWS_ROLE_ARN
environment variable to the deployment. Not sure if this is expected to be manually added, couldn't find anything to say either way
Thanks for pointing this out (and providing the solution)! It looks like setting that env variable is part of the aws setup, but either way i've labeled it so the docs team can take a look.
@kirkdave what version of EKS are you using? EKS should automatically inject the AWS_ROLE_ARN
into the pod as part of the IRSA stuff. I'm running vault on EKS 1.17 and 1.18 with IRSA and I have not ran into this issue (also deployed with Helm).
@kirkdave what version of EKS are you using? EKS should automatically inject the
AWS_ROLE_ARN
into the pod as part of the IRSA stuff. I'm running vault on EKS 1.17 and 1.18 with IRSA and I have not ran into this issue (also deployed with Helm).
I'm currently running into this issue on an EKS 1.17 cluster. Could I bother you to post your chart override values for Vault?
I've gone into detail at https://github.com/hashicorp/vault-helm/issues/368#issuecomment-761585670
I've tried now manually setting the AWS_ROLE_ARN. Attempting to init the vault looks like a step backward for me as it attempts to use the root role:
Error initializing: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:
* failed to store keys: failed to encrypt keys for storage: error encrypting data: AccessDeniedException: User: arn:aws:iam::472409228388:root is not authorized to perform: kms:Encrypt on resource: arn:aws:kms:eu-north-1:472409228388:key/260757fd-c5e8-49b4-972d-66fb5a366a1e
status code: 400, request id: 93afeb85-8540-4f56-9771-918245bcfea4
With the AWS_ROLE_ARN attached automatically, it looks like Vault attempts to use the correct role but without success:
* failed to store keys: failed to encrypt keys for storage: error encrypting dat
a: AccessDeniedException: User: arn:aws:sts::472409228388:assumed-role/vault/161
0984202537365570 is not authorized to perform: kms:Encrypt on resource: arn:aws:
kms:eu-north-1:472409228388:key/260757fd-c5e8-49b4-972d-66fb5a366a1e
status code: 400, request id: 1abb7c91-0f8c-4175-a745-7f103d07169a
This despite setting the correct policy permissions via Terraform:
data "aws_iam_policy_document" "vault" {
statement {
sid = "VaultKMSUnseal"
effect = "Allow"
actions = [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey",
]
resources = ["*"]
}
}
This turned out to be KMS key related. For some reason the original key just didn't want to allow permissions. Created a new key and boom, all is well
I am also facing such issue, i have tried by adding "AWS_ROLE_ARN", to the env. It doesn't consider it.
Also below error is for ServiceAccount.
eal.awskms: error assuming role: roleARN=* tokenPath=/var/run/secrets/eks.amazonaws.com/serviceaccount/token sessionName= err="WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: 9ce88500-9b64-4d6d-b98f-02608eba78c5"
Ran into a same error message on an older EKS-based Vault that's gone from evolution from KIAM to IRSA. In my case, the issue was that the KMS Key Policy only had the default policy (the one that allows Admin access). This worked in the KIAM environment. I had to explicitly add my IRSA role to the KMS Key Policy to get Vault to auto unseal with KMS when using IRSA. In hindsight, this makes sense. Had to change from:
{
"Version": "2012-10-17",
"Id": "key-default-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXXXXXX:root"
},
"Action": "kms:*",
"Resource": "*"
}
]
}
to:
{
"Version": "2012-10-17",
"Id": "key-default-1",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::XXXXXXXXXX:root",
"arn:aws:iam::XXXXXXXXXX:role/xxxx-irsa-vault-kms-role"
]
},
"Action": "kms:*",
"Resource": "*"
}
]
}
I got it working with Vault v1.14.1 The issue was in SA annotation. If you don't add the annotation "eks.amazonaws.com/role-arn" to the Vault's SA account then EKS won't mount token "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" to vault pod. Without that Vault ignores role_arn/web_identity_token_file in the configuration file
Definitively KO in 1.14.0, got it working in Vault v1.14.1 simply by changing the image tag. Every other resources (Service account, annotations, IAM role, KMS policy and so on) were unchanged between the two image tags
Describe the bug When running Vault on EKS (deployed via Helm chart) it is not using the IAM role annotated on the service account to get permissions for AWS API calls. It uses the IAM role assigned to the worker node
To Reproduce Steps to reproduce the behavior:
Expected behavior Vault should use the role annotated on the service account to authenticate with AWS and successfully describe the KMS key
Environment:
Additional context To validate the IAM role can be successfully assumed using IRSA I have run the following command, which was successful
Which returns