Closed towithyou closed 2 months ago
Just to add vault k8s auth token ttl 1 day vault secret db ttl and max_ttl 3 day Is the auth ttl time less than the secret ttl, which may have an impact when restarting the service?
Hi @towithyou, it looks like you are running an older VSO release, would you be able to upgrade to v0.6.0 (current latest) and see if the problem persists?
Thanks,
Ben
Thank you for your reply. Upgrading to the latest version(v0.6.0) still has issues after restart vault operator. I have learned from reviewing the source code that the connection of the vault client will be cached in memory. When the service restarts, a new vault client will be reinitialized. At this time, the previous vault auth ttl will automatically revoke the dynamic secret lease after it expires. My suggestion is to set the auth token ttl to be greater than or equal to the dynamic secret ttl. Is my understanding correct
@towithyou - I see, if you are syncing VaultDynamicSecrets you will need to enable client cache storage. When the storage is enabled new VSO leaders will pick up the token renewals on when a new election occur, or whenever a new VSO Pod is replaced. Please see https://developer.hashicorp.com/vault/docs/platform/k8s/vso/sources/vault/client-cache for more information.
I configured and solved this problem according to the document requirements. Thank you very much for your support
Describe the bug After restarting the vault operator, the lease will be automatically deleted at night. If the service is not restarted, long-term operation can renew the lease normally
Vault version vault operator 0.4.3 vault server 1.15.4
VDS yaml, Normally, it will expire at 2024-05-11 10:10:34
Application deployment:
Vault server logs 2024-05-08T23:13:10.305Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-08T23:23:10.354Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-08T23:33:10.469Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-08T23:43:10.521Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-08T23:53:10.568Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:03:10.617Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:03:41.059Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/hde2c354672e648b920ec415f5c271d9e15da25655a3d2daf9373b3455efa4798 2024-05-09T00:03:41.078Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/task-platform-rw/lONHRDFmJb3KHCoCa1kym3C1 2024-05-09T00:04:22.233Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/h78af65f30b7af74ce2b7230cf94717a9f673973239ef9c63d3d1d0a963d0f4e1 2024-05-09T00:04:22.241Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/workflow-rw/gmGCdiSReSYLAAUYbwzRfKyy 2024-05-09T00:13:10.665Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:23:10.715Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:28:29.110Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/service-tree-rw/wyYsCXdvlieoU7DPIObv0PZC 2024-05-09T00:28:29.120Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/h6f1a708218e6e2ab23d6e2d0c755666f63da531ff7eeb2b3d94ab12e99a51b1d 2024-05-09T00:28:29.123Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/service-tree-rw/yyCMX1FRYrnNbovVUFzVsNrq 2024-05-09T00:33:10.822Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:40:49.342Z [INFO] expiration: revoked lease: lease_id=auth/group-k8s/login/hc319ff3f9273e3b9f61d146053c0a3e4707ad29a6518776b1b13fa860d3b9d40 2024-05-09T00:43:10.868Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T00:53:10.918Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:03:10.963Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:13:11.012Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:13:37.046Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/db-manager-r/iJdtQ4lt7iI61jXr0EDCASs9 2024-05-09T01:23:11.054Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:33:11.178Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:43:11.225Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T01:53:11.277Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T02:02:49.405Z [INFO] expiration: revoked lease: lease_id=a-cloud-db/creds/resource-utilization-rw/3fbCmUPGxt59MSbIYPaDf6KG 2024-05-09T02:03:11.326Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T02:13:11.376Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms 2024-05-09T02:23:11.426Z [DEBUG] core.autoseal: seal wrapper health test passed: seal_name=awskms
Vault operator logs, From the logs, there are no obvious errors 2024-05-08T23:37:45Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286978772"}, "reason": "RolloutRestartTriggered"} 2024-05-09T00:23:12Z DEBUG events Lease renewal duration was truncated from 3600s to 682s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286977175"}, "reason": "SecretLeaseRenewal"} 2024-05-09T00:23:12Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/OZWMNxMvQSwTSqCmpItunb5d", horizon=50m13.530685162s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "SecretRotated"} 2024-05-09T00:23:12Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "RolloutRestartTriggered"} 2024-05-09T00:28:42Z DEBUG events Lease renewal duration was truncated from 3600s to 543s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286978772"}, "reason": "SecretLeaseRenewal"} 2024-05-09T00:28:42Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/71By43ksdbAkBiyx7GzLPfxw", horizon=53m49.031526697s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "SecretRotated"} 2024-05-09T00:28:42Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "RolloutRestartTriggered"} 2024-05-09T01:13:26Z DEBUG events Lease renewal duration was truncated from 3600s to 586s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287002316"}, "reason": "SecretLeaseRenewal"} 2024-05-09T01:13:26Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/s0pZr0mdRbGdyAM6Md5LWKGL", horizon=53m45.735179044s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "SecretRotated"} 2024-05-09T01:13:26Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "RolloutRestartTriggered"} 2024-05-09T01:14:53Z DEBUG events Lease renewal duration was truncated from 7200s to 928s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"286975094"}, "reason": "SecretLeaseRenewal"} 2024-05-09T01:14:53Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_workflow-rw/6zJP1ph0j6akcxD5Dwxl4LzT", horizon=1h38m55.068554041s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028899"}, "reason": "SecretRotated"} 2024-05-09T01:14:53Z DEBUG events Rollout restart triggered for {Deployment a-cloud-workflow} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-workflow","uid":"d34f9a84-1ac9-4eb8-9297-d963624763da","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028899"}, "reason": "RolloutRestartTriggered"} 2024-05-09T01:22:31Z DEBUG events Lease renewal duration was truncated from 3600s to 371s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287005035"}, "reason": "SecretLeaseRenewal"} 2024-05-09T01:22:32Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/oVcjMlwDDNPaKV0nNWblE5Rd", horizon=51m11.582480918s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "SecretRotated"} 2024-05-09T01:22:32Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "RolloutRestartTriggered"} 2024-05-09T02:07:11Z DEBUG events Lease renewal duration was truncated from 3600s to 375s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287028179"}, "reason": "SecretLeaseRenewal"} 2024-05-09T02:07:12Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_service_tree-rw/4gZA9LVKzopkRJw4qjgyRQCV", horizon=50m52.643550801s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287055823"}, "reason": "SecretRotated"} 2024-05-09T02:07:12Z DEBUG events Rollout restart triggered for {Deployment a-cloud-service-tree} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-service-tree","uid":"2be7fc7c-ae64-4c1b-96f9-5323fc5e925c","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287055823"}, "reason": "RolloutRestartTriggered"} 2024-05-09T02:13:43Z DEBUG events Lease renewal duration was truncated from 3600s to 529s, requesting new credentials {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287032970"}, "reason": "SecretLeaseRenewal"} 2024-05-09T02:13:43Z DEBUG events Secret synced, lease_id="group-database/creds/a_cloud_cmdb-rw/M6SR2Lw3xE5bXLQtHHAIdytq", horizon=51m20.519425641s {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287059288"}, "reason": "SecretRotated"} 2024-05-09T02:13:43Z DEBUG events Rollout restart triggered for {Deployment a-cloud-cmdb} {"type": "Normal", "object": {"kind":"VaultDynamicSecret","namespace":"devops-test","name":"a-cloud-cmdb","uid":"02515b7c-70ea-4f51-87cf-bbe2cf0ab612","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"287059288"}, "reason": "RolloutRestartTriggered"}