search

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
454 stars 95 forks source link

Vault Dynamic secret AWS not updated #275

Open sebglon opened 1 year ago

sebglon commented 1 year ago

Describe the bug Credential type: IAM_USER VSO: 0.1.0 Vault: 1.12.1

The VaultDynamicSecret is not renewed

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  creationTimestamp: "2023-06-20T07:20:35Z"
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: ubble-devops-workloads
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: minio-encrypted-aws-account
  namespace: minio
  resourceVersion: "618772646"
  uid: b57d6fd7-ca30-4afa-a5a6-6098304db2e9
spec:
  destination:
    create: true
    name: minio-encrypted-aws-account
  mount: aws
  path: creds/minio-encrypted-aws
  renewalPercent: 67
  rolloutRestartTargets:
  - kind: Deployment
    name: minio-encrypted-aws
  vaultAuthRef: minio-encrypted-aws

If i check on my AWS IAM, i have no user created. I check the k8s generated secret, i have an accessKey and secretKey but it not work. I have trying to restart VSO without success.

Here is the status of the VaultDynamicSecret after the restart:

Status:
  Last Generation:       1
  Last Renewal Time:     1687247954
  Last Runtime Pod UID:  52d14ca0-b6cf-4dec-aac2-60e2dd49787d
  Secret Lease:
    Duration:    2764800
    Id:          aws/creds/minio-encrypted-aws/xnbo15l2p5phfLsb7EL9X8Kn
    Renewable:   true
    Request ID:  c5505f06-1a05-87b3-f89f-8c0c1f1eaffe
  Static Creds Meta Data:
    Last Vault Rotation:  0
    Rotation Period:      0
    Ttl:                  0
Events:
  Type    Reason              Age    From                Message
  ----    ------              ----   ----                -------
  Normal  SecretLeaseRenewal  7m23s  VaultDynamicSecret  Not in renewal window after transitioning to a new leader/pod, lease_id=aws/creds/minio-encrypted-aws/xnbo15l2p5phfLsb7EL9X8Kn, horizon=574h41m59.877389462s

If i drop the K8s secret and the Dynamic secret and re-deploy it, the secret is well generated with a valid accessKey and secretKey.

To Reproduce Steps to reproduce the behavior:

  1. Deploy application with the following yaml file with the following VSO custom resources.
  2. Any custom resources used for your secrets.
  3. ...
  4. See error (vault-secrets-operator logs, application logs, etc.)

Application deployment:

# Paste your application deployment yaml and custom resources here.
# Be sure to scrub any sensitive values!

Other useful info to include: kubectl describe deployment <app> and kubectl describe <vso-custom-resource> <app> output.

Expected behavior I expect a Secret renewal with an IAM_USER AWS VaultDynamicSecret

Environment

benashz commented 1 year ago

Hi @sebglon , would you be able to provide any VSO error logs/events that you encountered? The last event seems to be normal after having restarted VSO.

Thanks,

Ben

sebglon commented 1 year ago

Event on a VaultDynamic secret:

Events:
  Type    Reason              Age   From                Message
  ----    ------              ----  ----                -------
  Normal  SecretLeaseRenewal  32m   VaultDynamicSecret  Not in renewal window after transitioning to a new leader/pod, lease_id=aws/creds/minio-encrypted-aws/J4vj9xXvzofXkjjojn21S0ob, horizon=526h23m1.501646813s

Vault-Secret-Operator log for the Dynamic secret:

"@timestamp",levelname,message,"kubernetes.pod.name","identification_id"
"Jun 28, 2023 @ 09:04:58.114","-","2023-06-28T07:04:58Z DEBUG   events  Not in renewal window after transitioning to a new leader/pod, lease_id=aws/creds/minio-encrypted-aws/J4vj9xXvzofXkjjojn21S0ob, horizon=526h23m1.501646813s {""type"": ""Normal"", ""object"": {""kind"":""VaultDynamicSecret"",""namespace"":""minio"",""name"":""minio-encrypted-aws-account"",""uid"":""a68f26b7-cd95-463e-a64e-dc999a27644a"",""apiVersion"":""secrets.hashicorp.com/v1beta1"",""resourceVersion"":""247454479""}, ""reason"": ""SecretLeaseRenewal""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 28, 2023 @ 09:04:58.101","-","2023-06-28T07:04:58Z INFO    Starting workers    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""worker count"": 100}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 28, 2023 @ 09:04:57.900","-","2023-06-28T07:04:57Z INFO    Starting EventSource    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""source"": ""kind source: *v1beta1.VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 28, 2023 @ 09:04:57.900","-","2023-06-28T07:04:57Z INFO    Starting Controller {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 27, 2023 @ 14:03:11.556","-","2023-06-27T12:03:11Z DEBUG   events  Not in renewal window after transitioning to a new leader/pod, lease_id=aws/creds/minio-encrypted-aws/J4vj9xXvzofXkjjojn21S0ob, horizon=545h51m53.387644401s    {""type"": ""Normal"", ""object"": {""kind"":""VaultDynamicSecret"",""namespace"":""minio"",""name"":""minio-encrypted-aws-account"",""uid"":""a68f26b7-cd95-463e-a64e-dc999a27644a"",""apiVersion"":""secrets.hashicorp.com/v1beta1"",""resourceVersion"":""246938628""}, ""reason"": ""SecretLeaseRenewal""}","vault-secrets-operator-controller-manager-5887658cd8-2wqrc","-"
"Jun 27, 2023 @ 14:03:11.544","-","2023-06-27T12:03:11Z INFO    Starting workers    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""worker count"": 100}","vault-secrets-operator-controller-manager-5887658cd8-2wqrc","-"
"Jun 27, 2023 @ 14:03:11.442","-","2023-06-27T12:03:11Z INFO    Starting EventSource    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""source"": ""kind source: *v1beta1.VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-2wqrc","-"
"Jun 27, 2023 @ 14:03:11.442","-","2023-06-27T12:03:11Z INFO    Starting Controller {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-2wqrc","-"
"Jun 27, 2023 @ 12:24:33.705","-","2023-06-27T10:24:33Z DEBUG   events  Not in renewal window after transitioning to a new leader/pod, lease_id=aws/creds/minio-encrypted-aws/J4vj9xXvzofXkjjojn21S0ob, horizon=547h0m25.539869437s {""type"": ""Normal"", ""object"": {""kind"":""VaultDynamicSecret"",""namespace"":""minio"",""name"":""minio-encrypted-aws-account"",""uid"":""a68f26b7-cd95-463e-a64e-dc999a27644a"",""apiVersion"":""secrets.hashicorp.com/v1beta1"",""resourceVersion"":""246817303""}, ""reason"": ""SecretLeaseRenewal""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 27, 2023 @ 12:24:33.692","-","2023-06-27T10:24:33Z INFO    Starting workers    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""worker count"": 100}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 27, 2023 @ 12:24:33.490","-","2023-06-27T10:24:33Z INFO    Starting EventSource    {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret"", ""source"": ""kind source: *v1beta1.VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"
"Jun 27, 2023 @ 12:24:33.490","-","2023-06-27T10:24:33Z INFO    Starting Controller {""controller"": ""vaultdynamicsecret"", ""controllerGroup"": ""secrets.hashicorp.com"", ""controllerKind"": ""VaultDynamicSecret""}","vault-secrets-operator-controller-manager-5887658cd8-6fsx2","-"

Every day we restart the VSO pods but it not try to reconcile it. On my AWS console the IAM user is not re-created. it seems that we have de synchro between the AWS IAM lifecycle and the Vault Secret lifecycle. The AWS IAM user seems deleted before the end of the renewal period.

Here is the aws role:

{
   "request_id":"d8f6e77c-f671-ebff-4d8b-53d1fa75c720",
   "lease_id":"",
   "renewable":false,
   "lease_duration":0,
   "data":{
      "credential_type":"iam_user",
      "default_sts_ttl":0,
      "iam_groups":null,
      "iam_tags":null,
      "max_sts_ttl":0,
      "permissions_boundary_arn":"",
      "policy_arns":null,
      "policy_document":"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":[\"s3:ListAllMyBuckets\",\"s3:GetBucketLocation\"],\"Effect\":\"Allow\",\"Resource\":\"arn:aws:s3:::*\",\"Sid\":\"\"},{\"Sid\":\"AllowStatementBackup\",\"Action\":[\"s3:ListBucket\",\"s3:ListBucketVersions\",\"s3:GetBucketVersioning\"],\"Effect\":\"Allow\",\"Resource\":[\"arn:aws:s3:::backups.devops.env.ubble.ai\"],\"Condition\":{\"StringLike\":{\"s3:prefix\":[\"mongodb-backup-devops-a/*\",\"mongodb-backup-devops-a\",\"spilo/*\",\"spilo\"]}}},{\"Sid\":\"AllowStatementAssets\",\"Action\":[\"s3:ListBucket\",\"s3:ListBucketVersions\",\"s3:GetBucketVersioning\"],\"Effect\":\"Allow\",\"Resource\":[\"arn:aws:s3:::misc.env.ubble.ai\",\"arn:aws:s3:::devops.env.ubble.ai\",\"arn:aws:s3:::public.env.ubble.ai\",\"arn:aws:s3:::stats.devops.env.ubble.ai\",\"arn:aws:s3:::ml.env.ubble.ai\"]},{\"Action\":[\"s3:PutObjectTagging\",\"s3:PutObject\",\"s3:GetObjectVersion\",\"s3:GetObjectTagging\",\"s3:GetObjectAttributes\",\"s3:GetObject\",\"s3:DeleteObjectVersion\",\"s3:DeleteObjectTagging\",\"s3:DeleteObject\",\"s3:AbortMultipartUpload\"],\"Effect\":\"Allow\",\"Resource\":[\"arn:aws:s3:::backups.devops.env.ubble.ai/mongodb-backup-devops-a/*\",\"arn:aws:s3:::backups.devops.env.ubble.ai/spilo/*\",\"arn:aws:s3:::misc.env.ubble.ai/*\",\"arn:aws:s3:::devops.env.ubble.ai/*\",\"arn:aws:s3:::public.env.ubble.ai/*\",\"arn:aws:s3:::stats.env.ubble.ai/*\",\"arn:aws:s3:::ml.env.ubble.ai/*\"],\"Sid\":\"AllowStatementObjects\"}]}",
      "role_arns":null,
      "user_path":""
   },
   "wrap_info":null,
   "warnings":null,
   "auth":null
}

And the AWS root config: 

{
   "request_id":"7f34316b-8c02-5e24-6e0a-fa50e68f4fdd",
   "lease_id":"",
   "renewable":false,
   "lease_duration":0,
   "data":{
      "access_key":"xxxxxx",
      "iam_endpoint":"",
      "max_retries":-1,
      "region":"eu-west-1",
      "sts_endpoint":"",
      "username_template":"{{ if (eq .Type \"STS\") }}{{ printf \"vault-devops-a-%s-%s\"  (unix_time) (random 20) | truncate 24 }}{{ else }}{{ printf \"vault-devops-a-%s-%s-%s\" (printf \"%s-%s\" (.DisplayName) (.PolicyName) | truncate 34) (unix_time) (random 20) | truncate 56 }}{{ end }}"
   },
   "wrap_info":null,
   "warnings":null,
   "auth":null
}
sebglon commented 1 year ago

After setting a lease with cmd:

curl \
    --header "X-Vault-Token: ..." \
    http://127.0.0.1:8200/v1/aws/config/lease

with file:

{
  "data": {
    "lease": "30m0s",
    "lease_max": "12h0m0s"
  }
}

Before this change the lease was set by default to 0. Vault destroy the iam_user if it is not used but i have not found where is defined the clean time interval. On this case, the VSO does not detect the iam_user deletion and it not re-create it.

sizgiyaev commented 1 year ago

Is there any news about the issue? We are also experiencing the same. It makes VSO pretty useless when using AWS dynamic secrets.

benashz commented 1 year ago

Hi @sizgiyaev , just wanted to let you know that we are currently working this issue. It should go out in the next VSO release. No ETA yet for that, though.

Thanks,

Ben

benashz commented 1 year ago

So we took a closer look at this issue. Essentially, VSO does not yet support notification on updates to a Vault Secret engine's backend configuration. It can only rely on the TTL associated with the last sync'd secret's lease. There are few potential options to address the issue:

  1. Add support for polling Vault for secret backend config changes
  2. Integrate with Vault's notification system which could (does not yet) provide VSO with a "TTL config updates" notification.

We are planning to add Vault notification support to VSO soonish. Initially it will watch for lease revocations and updates to KV secrets. Upon lease revocation VSO will auto re-sync the VaultDynamicSecret (VDS). That gives the Vault admin the ability to trigger a VSO sync directly from Vault. Any VDS queued for reconciliation will be immediately synced.

I will sync up with the team to see how viable option 2 could be. Option 1 could also work but that would mean that VSO would need access to the various Vault config endpoints.

For the now it is recommended to ensure that the AWS secret engine is configured with a relatively short default-lease-ttl ~10m. VSO will ensure that the lease is renewed. This should be done before applying your VDS CR.

E.g:

$ vault secrets tune -default-lease-ttl=10m aws
Success! Tuned the secrets engine at: aws/

$ vault read sys/mounts/aws/tune
Key                  Value
---                  -----
default_lease_ttl    10m
description          n/a
force_no_cache       false
max_lease_ttl        768h
benashz commented 1 year ago

Of note I was unable to reproduce the issue where the AWS secret's engine did not properly create the AWS IAM user after receiving a request from VSO.

benashz commented 1 year ago

Here's what we would expect to see in the nominal condition for a AWS creds sync with a 15m default-lease-ttl (5 VDS instances):

$ kubectl get events -n demo-ns --show-kind  --sort-by '{.lastTimestamp}'

LAST SEEN   TYPE     REASON               OBJECT                                    MESSAGE
60m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m48.938029163s
52m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=10m25.960902897s
50m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-3   Renewed lease, lease_id=aws/creds/app/LDmu5PQNtD8IxKpbtGQCpPed, horizon=10m19.187986009s
50m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-2   Renewed lease, lease_id=aws/creds/app/q8YWaNwpJzwPL9bsWrx8y7gk, horizon=10m46.95307484s
50m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-4   Renewed lease, lease_id=aws/creds/app/u03Fj9fcxCtlt63bPIWl0zVn, horizon=10m11.780117153s
50m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m33.205819754s
42m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=10m47.982121274s
40m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-3   Renewed lease, lease_id=aws/creds/app/LDmu5PQNtD8IxKpbtGQCpPed, horizon=10m39.655669811s
40m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-4   Renewed lease, lease_id=aws/creds/app/u03Fj9fcxCtlt63bPIWl0zVn, horizon=10m4.869515982s
39m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-2   Renewed lease, lease_id=aws/creds/app/q8YWaNwpJzwPL9bsWrx8y7gk, horizon=10m34.31895765s
39m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m9.093865682s
31m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=10m17.669500916s
30m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-4   Renewed lease, lease_id=aws/creds/app/u03Fj9fcxCtlt63bPIWl0zVn, horizon=10m17.475555064s
29m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-3   Renewed lease, lease_id=aws/creds/app/LDmu5PQNtD8IxKpbtGQCpPed, horizon=10m47.486502871s
29m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m37.412810139s
29m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-2   Renewed lease, lease_id=aws/creds/app/q8YWaNwpJzwPL9bsWrx8y7gk, horizon=11m17.133685072s
21m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=10m10.232518959s
19m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-4   Renewed lease, lease_id=aws/creds/app/u03Fj9fcxCtlt63bPIWl0zVn, horizon=10m5.247458273s
19m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-3   Renewed lease, lease_id=aws/creds/app/LDmu5PQNtD8IxKpbtGQCpPed, horizon=10m41.123003526s
18m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m49.393596923s
17m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-2   Renewed lease, lease_id=aws/creds/app/q8YWaNwpJzwPL9bsWrx8y7gk, horizon=10m19.291103612s
11m         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=10m11.801716384s
9m45s       Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-4   Renewed lease, lease_id=aws/creds/app/u03Fj9fcxCtlt63bPIWl0zVn, horizon=10m26.557034835s
8m26s       Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-3   Renewed lease, lease_id=aws/creds/app/LDmu5PQNtD8IxKpbtGQCpPed, horizon=10m48.972022003s
8m          Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-0   Renewed lease, lease_id=aws/creds/app/gy1VCM1zoWTnZv1LqYMPjmhG, horizon=10m44.660998791s
7m34s       Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-2   Renewed lease, lease_id=aws/creds/app/q8YWaNwpJzwPL9bsWrx8y7gk, horizon=10m9.190137726s
50s         Normal   SecretLeaseRenewal   vaultdynamicsecret/vso-demo-aws-creds-1   Renewed lease, lease_id=aws/creds/app/SdaavaFRI8p1V2UCf1otqUA1, horizon=11m13.812898064s

If we count up the number of IAM users created for each VDS instance the result should be 5:

$ PAGER= aws iam list-users --query 'length(Users[?starts_with(UserName, `vault-demo`) == `true`])'

5

or

$ PAGER= aws iam list-users --query 'Users[?starts_with(UserName, `vault-demo`) == `true`].UserName'

[
    "vault-demo-auth-mount-demo-ns-default-app-1689365110-PZp3URGsaRs",
    "vault-demo-auth-mount-demo-ns-default-app-1689365132-6LpGpt9W8M2",
    "vault-demo-auth-mount-demo-ns-default-app-1689365221-a1Qzdi49GPX",
    "vault-demo-auth-mount-demo-ns-default-app-1689365233-FfvCUc4s7ag",
    "vault-demo-auth-mount-demo-ns-default-app-1689365263-y5KdY7iGKYU"
]
benashz commented 1 year ago

If it is not possible to configure the value of the secret mount's default-lease-ttl, you can tune/lower the value of the VaultDynamicSecretSpec's renewalPercent, which will cause VSO to poll more frequently for expired leases.

adrianmoisey commented 1 year ago

2. Integrate with Vault's notification system which could (does not yet) provide VSO with a "TTL config updates" notification.

Can you expand a little on this option? Mostly... what is Vault's Notification system? I've been using Vault for some time and never really come across it. I tried searching the docs and also can't find anything. Could you point me in the right direction?

sizgiyaev commented 11 months ago

Any updates about the resolution here?